Exporting REDCap Data into R

1. Get a REDCap account for a project URL and API key

2. Go to https://reccap.cloudapps.unc.edu/accounts/login

3. Click “New Application” and choose Blackbalsam Clinical

4. Click on “black balsam” folder and then click on RedCapAPIRDemo.ipynb

5. In the notebook, on cell 2 substitute REDCap URL and API key with your credentials

6. Run first three cells in the notebook

7. To export records, use show_forms, show_fields, or show_all_records cells

Using the restartr notebook to get data into mongodb

1. Go to https://reccap.cloudapps.unc.edu/accounts/login and login.

2. Click on “New Application”, choose Blackbalsam Clinical then click "Create Application".

3. Click on the "blackbalsam" directory and then launch the RestartrApiDemo.ipynb notebook.

4. In cells 3 to 6 substitute your api-key here "<put-api-key>".

5. Run the first two cells in the notebook.

• The first cell imports the required dependencies.

• The second cell is where data can be formatted to insert into mongodb.

6. Run the third cell to persist data into mongodb.

• This cell will call the observation api and insert data into mongodb.

• Upon a successful call it will return an id, for example {"id": "\"5f355dd39e55e62921768dec\""}.

7. To query the data use, query by "_id", query by sub-field, or just use query methods provided in the RestartrApiDemo.ipynb notebook.

Introduction

Jupyter/datascience includes popular packages for data analysis from the Python, Julia and R communities and also packages are included from its ancestor images jupyter/sci-py notebook, jupyter/r-notebook and jupyter/minimal-notebook.

Some of the packages it includes are,

dask, pandas, numexpr, matplotlib, scipy, seaborn, scikit-learn, scikit-image, sympy, cython, patsy, statsmodel, cloudpickle, dill, numba, bokeh, sqlalchemy, hdf5, vincent, beautifulsoup, protobuf, xlrd, bottleneck, and pytables packages

ipywidgets and ipympl for interactive visualizations and plots in Python notebooks.

Facets for visualizing machine learning datasets.

The Julia compiler and base environment.

IJulia to support Julia code in Jupyter notebooks.

HDF5, Gadfly, and RDatasets packages.

Working with jupyter-datascience notebook in HeLx

Step-1:

Launch a jupyter-datascience notebook from HeLx by clicking on “Launch Application” button.

Step-2:

This brings us to the jupyter-lab panel where we can select the environment that we want to work on (Python, Julia, R).

Step-3:

Start working on it. Below code shows loading iris dataset (features, labels) from sklearn package to train/test our machine learning model.

Allocating Resources

Navigate to the Active tab under Workspaces. You will see a list of your currently running Applications.

Find the running app whos resources you want to modify and click the Update button.

A popup window will appear.

You can modify the allocated CPUs as well as how much memory is going to your app. Click Apply to save your changes and modify your app.

Renaming an App

To help identify one app from another, use the Update button to give your running Apps unique nicknames. This can help you keep track of the various app instances you have running.

Follow the steps above for modifying resources to change a Workspace Name. Click Apply once you are done.

Deleting an App

Once you have finished using an app, you can delete it from your workspace by clicking the Trash Button for that App. This helps free up App resources as well as keep your Active page less cluttered.

You can similarly delete all running apps by clicking Stop App. This will end all running apps in your Active page.

Testing the CloudTop Docker

Step 1: Run the CloudTop Docker.

USER_NAME and VNC_PW can be whatever you want: those are the authentication info you will need to log in for Step 2. Change the tag to whichever tag you want to test.

Step 2: Connect to the running docker

• Browse to localhost:8080

• Enter the USERNAME and VNC_PW you specified when starting the Dockerfile

• Press the Login button

• Wait for Guacamole to respond

Step 3: Make sure the home directory is OK

• Start a terminal emulator from the applications menu. In the resultant shell, type

• You should see /home/USER_NAME where USER_NAME is the user name specified in Step 1

• Note the presence of the Firefox browser icon

At this point the basic CloudTop functionality is working.

Testing the CloudTop OHIF Docker

Step 1: Run the CloudTop OHIF docker:

where OUR_CLIENT_ID is found in the file.

google_health.env file in the renci_data_stage directory of our keybase account. USER_NAME and VNC_PW can be whatever you want: those are the authentication info you will need to log in for Step 2. Change the tag to whichever tag you want to test.

Step 2: Connect to the running docker

• Browse to localhost:8080

• Enter the USERNAME and VNC_PW you specified when starting the Dockerfile

• Press the Login button

• Wait for Guacamole to respond

Step 3: Make sure the home directory is OK

• Start a terminal emulator from the applications menu. In the resultant shell, type:

• You should see /home/USER_NAME where USER_NAME is the user name specified in Step 1

• Note the presence of the Firefox browser icon

At this point the basic CloudTop functionality is working.

Step 4: Test the OHIF functionality

• Exit the terminal emulator by typing “exit”

• Click the Firefox icon and browse to localhost:3000

• At this point you will be prompted for your Google user ID.

• Click Next.

• Google may prompt you to choose the account you wish to proceed with. If prompted, pick your G Suite account.

• Click Next.

• Enter your password. The browser will ask if you want to save the password. It doesn’t matter if you do or not

• Respond to the 2 step authentication. If you haven't used it before, you may be prompted to set up the 2 step authentication.

• You should now see the basic OHIF screen with a large selection of projects.

Step 5: Browse to Your Data Set

• Select helx-dev

• Select the northamerica- northeast1 region

• Select the DicomTestData dataset

• Select the TestData Dicom Store

• You should now see our test datasets. Chose your test data set and have fun!

Testing the CloudTop ImageJ/Napari Docker

Step 1: Start the Docker

• Start the docker with the following command:

where USER_NAME and VNC_PW can be whatever you want: those are the authentication info you will need to log in for Step 2. Change the tag to whichever tag you want to test.

Step 2: Connect to the running docker

• Browse to localhost:8080

• Enter the USERNAME VNC_PWyou specified when starting the Dockerfile

• Press the Login button

• Wait for Guacamole to respond

Step 3: Make sure the home directory is OK

• Start a terminal emulator from the applications menu. In the resultant shell, type:

• You should see /home/USER_NAME where USER_NAME is the user name specified in Step 1

• Note the presence of the ImageJ, Napari and Firefox browser icon. If any are missing the test fails.

At this point the basic CloudTop functionality is working. Next we will want to verify that ImageJ and Napari are working

Step 4: Make sure the ImageJ application launcher works correct

• Exit the terminal application and click the ImageJ icon. There is no ImageJ test data included in the docker.

• Exit ImageJ and make sure the Napari application launcher works correctly.

• The docker does not contain any test data. The docker test is now complete.

• Exit Napari and stop the docker.

HeLx puts the most advanced analytical scientific models at investigator's finger tips using equally advanced cloud native, container orchestrated, distributed computing systems.

HeLx can be applied in many domains. Its ability to empower researchers to leverage advanced analytical tools without installation or other infrastructure concerns has broad reaching benefits.

For a more technical overview of HeLx and it's components, view this user guide.

Introduction

Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages.

Its fluent DSL simplifies the implementation and the deployment of complex parallel and reactive workflows on clouds and clusters.

Working with Nextflow on HeLx

Step-1:

Launch a Nextflow API by clicking on the "New Application" button on the app manager page on HelX.

Step-2:

This brings us to the Nextflow API home page, where we can view the launched workflows and create new workflows.

Step-3:

Below is a demo of how to launch a systemsgenetics/kinc workflow. Click on "Create Workflow" button and fill in the form to give it a "Name" and specify the Pipeline (in this case systemsgenetics/kinc-nf).

Step-4:

Uploading the necessary files, a GEM file in the format "*.emx.txt" and a nextflow.config file(can upload all files at once). Click on "Upload" button.

Step-5:

Now we are all set to launch the workflow. Go ahead and click on "Launch" button. This should show all the logs of the processes/jobs running in the background on the Kubernetes cluster.

Coming soon.

Running the DICOM Viewer Application

Step 1: Connect to the application

• Enter the USERNAME and VNC_PW you have been provided

• Press the Login button

• Wait for the app to respond

Step 2: Use OHIF functionality

• Click the Firefox icon and browse to localhost:3000

• At this point you will be prompted for your Google user ID.

• Click Next.

• Google may prompt you to choose the account you wish to proceed with. If prompted, use the account with which you logged in to the App Store

• Click Next.

• Enter your password. The browser will ask if you want to save the password. It doesn’t matter if you do or not

• Respond to the 2 step authentication. If you haven't used it before, you may be prompted to set up the 2 step authentication.

• You should now see the basic OHIF screen with a large selection of projects.

Step 3: Browse to The example Data Set

• Select helx-dev

• Select the northamerica- northeast1 region

• Select the DicomTestData dataset

• Select the TestData Dicom Store

• You should now see our test datasets. Chose your test data set and have fun!

HeLx Workspaces are deployed to Kubernetes in production using Helm. The main deployment concerns are:

• Security: Secrets are added to the container via environment variables.

• Persistence: Storage must be mounted for a database.

Process

Follow these instructions for your HeLx install.

Coming soon.

Q FOR HINA: do we think technical documentation should be linked

Additional technical documentation can be found here.

Log in and Start your App

Step-1: Begin by navigating to your team's instance of HeLx and signing in using either GitHub or Google.

Step-2: Navigate to Workspaces in the top nav bar.

Step-3: Select the App you wish to launch. Modify any resources or begin with the base settings (note: all resources can be modified after an App has successfully launched).

You can learn more about any HeLx App by clicking the About button on an App's card.

Click Launch when you are ready to launch your Application.

HeLx can be applied in many domains. It's ability to empower researchers to leverage advanced analytical tools without installation or other infrastructure concerns has broad reaching benefits.

EduHeLx

EduHeLx was developed to address the needs of courses with programming components and currently supports programming using Python and R. Previously, students were required to download a course’s programming software onto their own computers, and instructors had to work one-on-one with students to troubleshoot issues throughout the semester; this was so time-consuming that it took away from teaching time and derailed course schedules, especially in computer science courses with more than 250 students.

TraCS

The OMOP Research Data Repository (Deidentified) project (ORDR-D) is a proposed to solution for a number of challenges facing researchers:

• Ever-increasing demand for data

• Quick-turnaround requests with complex needs and hard deadlines

• Lack of infrastructure to support researchers wanting to work with large amounts of data

• Desire to "tinker" with data post-provisioning ("Just one more thing...")

HeLx supports the user functionality and system functionality for ORDR-D. This includes deidentified CDWH data in OMOP format to the RENCI Secure Enclave (hosted at ITS Manning), a data environment that has been approved for HIPAA data; management of the user interface that enables users to interact with/analyze this data; and providing UNC users with the ability to apply for access on a project-by-project basis.

READDI-AC

The Rapidly Emerging Antiviral Drug Development Initiatives AViDD Center (READDI-AC) platform is a response to the Emerging Viruses (EV) constantly arising from zoonotic reservoirs to threaten global health and economic stability, as illustrated by the ongoing COVID19 pandemic.

READDI-AC integrates cutting-edge discovery-based academic science with product development-oriented industrial drug development transforming the historically reactive landscape of emerging virus drug development to one that is proactive.

A component of READDI-AC is the data management core, relying on HeLx for the ingestion of multiple types of data.

BRAIN-I

BRAIN-I investigators create large images of brain tissue which are then visualized and analyzed using a variety of tools which, architecturally, are not web applications but traditional desktop environments. These include image viewers like ImageJ and Napari. Appstore presents these kinds of workspaces using CloudTop, a Linux desktop with screen sharing software and adapters for presenting that interface via a web browser. CloudTop allows us to create HeLx apps for ImageJ, Napari, and other visualization tools. These tools would be time consuming, complex, and error prone, for researchers to install and would still require them to acquire the data. With CloudTop, the system can be run colocated with the data with no installation required.

SciDAS

The Scientific Data Analysis at Scale project brings large scale computational workflow for research to cloud and on premise computing. Using the appstore, users are able to launch Nextflow API, a web based user interface to the Nextflow workflow engine. Through that interface and associated tools, they are able to stage data into the system through a variety of protocols, execute Nextflow workflows such as the GPU accelerated KINK workflow. Appstore and associated infrastructure has run KINK on the Google Kubernetes Engine and is being installed on the Nautilus Optiputer.

ReCCAP

RestartingResearch is a research project that is used for data management and analytics of COVID-19 RT-PCR and custom Antibody tests of UNC researchers using on-campus laboratories and facilities. ReCCAP uses Blackbalsam and Jupyter Data Science Jupyter notebooks in Python and R kernels as environments for data science experimentation. For more information, see the blackbalsam documentation.

HeLx 2.0

UI Front End

• Redesign inactive user session timeouts

• Update 404 error page for users

• Redesign sign in and sign out buttons and language

• Handle OAuth2 and SAML authentication

• Environment look and feel for each client/use case is customizable based on a server-side configuration specifying the project-specific “context” or product name including colors, logos, and front page text

App Interaction

• Update Django version to 3.1.6

• Allow naming apps when configuring and after launch

• Redesign app creation language

• Allow re-configuring app resources

• Setup and link to app resource docs that explain vertical scaling versus horizontal scaling, how it works, instance impacts, etc

• Lock resources which cannot be modified by the user

• Add a splash screen for polling and redirecting to the app instance

• Browse available applications based on the Tycho app registry metadata catalog

• Update Appstore README

Search

• Update the URL to allow users to share links to searches

• Increase the number of search results presented to users

• Handle TRAPI 1.0

• Make changes to Conceptto increase meaning for users

Stuck?

Contact HeLx Help with questions.