Data engineer journey ===================== This chapter follows a data engineer in Security Center: from the dashboard to publishing data, asking for access and deciding on requests from other teams. Terms are explained in :doc:`concepts`. How you create tables and read data in a notebook is in :doc:`notebooks`. The dashboard ------------- Open ``https://.security.datafoundation.eu`` and sign in (see :doc:`introduction`). You land on the **Dashboard**: .. image:: /_static/img/user-manual/data-engineer/01-dashboard.png :alt: The Security Center dashboard of a team member with the blocks My teams and My products :width: 700px - **My teams**: the teams you are a member of. Choose a team to see its members and its permissions. - **My products**: the data products your teams own. - **Pending approvals on my datasets**: requests of other teams for access to your products, waiting for your decision. See :ref:`Deciding on a request `. - **My permissions**: the access your teams have to products of others. The menu with the three dots at the top right offers **New catalog** and **Create team**: .. image:: /_static/img/user-manual/data-engineer/02-dashboard-menu.png :alt: The dashboard with the menu open: New catalog and Create team :width: 700px Browsing the catalog -------------------- Choose **Catalog** in the top bar. The tree on the left shows the catalogs; open one to see its schemas and tables. Next to an item you see what you may do with it: **owner**, **writer**, **reader** or **filtered**. Everybody can use the platform catalog ``main``. .. image:: /_static/img/user-manual/data-engineer/03-catalogus-tabel.png :alt: The Catalog page with the tree on the left and the details of the table finance.facturen.regels :width: 700px - Type in **Filter by name** to narrow the tree. Use the arrow keys to move through the tree and Enter to open an item. - Select an item to see its details: the path, the owner team (**Product**), the format (for example Delta, with the number of columns) and the line **You have owner through a grant on finance**, which tells you what you may do and where that comes from. - The tab **Access** shows the **Access trail**: where access is given along the path from the catalog down to the item, and **Who has access**: every team and role, where it comes from (**Origin**) and who approved it. - The tab **Columns** lists the columns with their types. - The tab **Manage** is only there for owners, see :ref:`Managing your data product `. .. image:: /_static/img/user-manual/data-engineer/04-kolommen.png :alt: The tab Columns of a table, listing five columns with their types :width: 700px Creating a data product ----------------------- A data product starts as a catalog that your team owns. Choose **New catalog** (on the dashboard or in the catalog tree): 1. Fill in the **Catalog** (the name, for example ``finance``) and choose the **Owner (team)**. A name has lower-case letters, digits and ``_`` and starts with a letter; capitals you type become lower case, because Spark and Trino always use lower-case names. If the dialog also shows **Kind**, choose **Product catalog**, see :doc:`concepts`. Add a **Contact**, a **Classification** and an **Description**, and tell the administrator why you need it in **Reason of request**. Leave **Product name** empty to use the name of the catalog. .. image:: /_static/img/user-manual/data-engineer/05-nieuwe-catalogus.png :alt: The New catalog dialog with a catalog name, an owner team, a description and a reason :width: 700px 2. Choose **Save**. With the default policy the dialog says that this becomes a request: an administrator approves it before it is created. You see the message **Request submitted**. .. image:: /_static/img/user-manual/data-engineer/06-aanvraag-ingediend.png :alt: The dashboard with the message Request submitted :width: 700px 3. When the administrator has approved (see :doc:`admin`) the catalog appears in the tree with the label **owner**. .. image:: /_static/img/user-manual/data-engineer/07-catalogus-finance.png :alt: The catalog finance in the tree with the label owner and its details :width: 700px You create the schemas and tables in the catalog with Spark in a notebook, see :doc:`notebooks`. They appear in the tree and belong to your team. Requesting access to data ------------------------- When you select a table, schema or catalog you may not use yet, the page says **You have no access** and offers **Request access**. .. image:: /_static/img/user-manual/data-engineer/08-geen-toegang.png :alt: A table for which the page says You have no access, with the button Request access :width: 700px 1. Choose **Request access**. The dialog says in words what you ask for, for example *this one table*. Choose the button below it if you want to ask for the whole schema, including everything that is added later. .. image:: /_static/img/user-manual/data-engineer/09-toegang-aanvragen.png :alt: The Request access dialog with a contact, a team, a role and a reason :width: 700px 2. Pick the **Team** that asks and the **Role** you need (**Reader**, **Writer** or a custom role of the owner, for example ``regio_noord``). Give a **Reason of request** and choose **Save**. 3. You see **Request send**. The owner team sees your request and decides. Until then nothing changes for you. .. image:: /_static/img/user-manual/data-engineer/10-aanvraag-verzonden.png :alt: The table page with the message Request send :width: 700px .. _um-deciding: Deciding on a request --------------------- Requests for your products show up in two places: on the dashboard under **Pending approvals on my datasets**, and on the tab **Manage** of the product (the number on the tab tells you how many wait). .. image:: /_static/img/user-manual/data-engineer/11-permissie-aanvragen.png :alt: The dashboard with the block Pending approvals on my datasets listing a request from another team :width: 700px 1. Open the product and choose the tab **Manage**. **Pending requests** lists the team, the role, the scope (**Scope**), the reason (**Comment**) and the contact of the requester. .. image:: /_static/img/user-manual/data-engineer/12-aanvraag-beheer.png :alt: The tab Manage with a request from Team Marketing under Pending requests :width: 700px 2. Choose **Approve** or **Reject**. A window shows the request again; give a reason and choose **Save**. The requester sees your reason. .. image:: /_static/img/user-manual/data-engineer/13-aanvraag-goedkeuren.png :alt: The approval dialog showing the team, the role, the requester and the scope, with a reason field :width: 700px 3. On the tab **Access** the team now appears under **Who has access**, with your name as the one who approved it. **Revoke** takes the access away again. .. image:: /_static/img/user-manual/data-engineer/14-toegang-na-goedkeuring.png :alt: The tab Access with Team Marketing having filtered access, approved by an owner, with Revoke :width: 700px Rejected and revoked requests stay visible under **Rejected and revoked**, with the date and the reason. .. _um-managing: Managing your data product -------------------------- As an owner you see the tab **Manage** on your catalogs, schemas and tables. Besides the requests it holds the roles, the row filters and the column masks of the product. A custom role ~~~~~~~~~~~~~ Besides **reader** and **writer** you can make your own roles. Choose **Add role**, enter a **Role name** (role name: lower case, starting with a letter, for example ``regio_noord``) and choose **Save**. A new role belongs to the whole product, not to one table. It is available at once as a role to request and as a role to give a filter or a mask. .. image:: /_static/img/user-manual/data-engineer/15-rol-toevoegen.png :alt: The Add role dialog with the role name regio_noord :width: 700px A row filter ~~~~~~~~~~~~ A row filter hides rows. Choose **Add row filter** and fill in: - **Role**: the role the rule applies to. - **Column**: the column to test. - **Operator** and **Value**: for example ``=`` and ``noord``. A row is visible when all conditions of the rule match; **Add another predicate** adds another condition. Several rules for the same role are combined with OR. .. image:: /_static/img/user-manual/data-engineer/16-rijfilter.png :alt: The Add row-filter rule dialog with the role regio_noord, the column regio and the value noord :width: 700px A column mask ~~~~~~~~~~~~~ A column mask hides the values of a column. Choose **Add column mask**, pick the **Role**, the **Column name** and the **Mask type**. The dialog shows an example of the result: .. list-table:: :header-rows: 1 * - Mask type - Effect * - ``full_redact`` - Replaces the value completely. * - ``hash`` - Replaces the value by a one-way hash. * - ``partial_mask`` - Masks a part and keeps the rest visible, for example only the last four characters. * - ``null`` - Returns an empty value (NULL). .. image:: /_static/img/user-manual/data-engineer/17-kolommasker.png :alt: The Add column-mask rule dialog with the role regio_noord, the column email and the mask full_redact :width: 700px Under **Row filters and column masks** you see all rules of the table. **Remove** removes a rule. .. image:: /_static/img/user-manual/data-engineer/18-filters-overzicht.png :alt: The tab Manage with a row filter regio = noord and a column mask on email for the role regio_noord :width: 700px .. warning:: A team that holds a role with a filter or a mask reads the table through Trino, with the filter and the mask applied. The team can read it, also in a notebook, but cannot write to it. Give a team a **writer** role if it has to change the data. Sharing data with another team ------------------------------ You do not have to wait for a request: as an owner you can give another team a role yourself. Choose **Share** on a catalog, schema or table, pick the **Team**, the **Role** and a **Contact**, give a reason and choose **Save**. To take access away again choose **Revoke** next to the team on the tab **Access**. .. image:: /_static/img/user-manual/data-engineer/19-delen.png :alt: The Delen dialog with a team, a role, a contact and a reason :width: 700px