Python Example Reading From Google Cloud Storage

Learn how to admission the data you need from your Google Cloud Storage in a few easy steps. This guide will tell you how to configure and connect to your information source and provide details about setting various permissions.

Data Source Nuts

All information sources have a protocol and a label that you volition apply to reference your data. For instance Google Cloud Storage is the protocol we'll use in this guide and the label volition be automatically assigned to your information connectedness as a unique identifier, but you may modify it later if you wish.

Configure a New Data Connection to your Google Cloud Storage

To create a new data connection outset navigate to Algorithmia's Data Portal where you'll detect there is a drop down that says 'New Information Source' where y'all'll run into the options:

Select 'Google Deject Storage' and a grade volition open up to configure a connectedness. Here you volition demand to enter your Google Deject Storage credentials for the Service Account you lot volition be using.

For programmatic access to Google Deject Storage, you will demand to create a Service Account and associated primal. You lot can learn more in the Google Cloud documentation pages for Creating and managing service accounts and Creating and managing service account keys. The documentation also includes information virtually best practices for working with service accounts.

Yous tin copy your private_key_id, private_key, and client_email from the JSON credentials file generated when you select "create central" in your service business relationship.

NOTE: While an algorithm NEVER sees credentials used to access data in Google Cloud Storage, information technology is recommended that you provide admission that:

• Can only listing, become, and put objects to Google Deject Storage (i.east. cannot perform other operations on your account)
• Can simply access the paths in Google Cloud Storage that you desire Algorithmia to admission

Setting Labels For Data Connections

You volition need to provide a unique label for your data connector, editable in the "Label" field.

We crave these unique labels considering you lot may want to add multiple connections to the aforementioned Google Cloud Storage account and they volition each need a unique label for afterwards reference in your algorithm. The reason y'all might want to have multiple connections to the same source is then you can set different admission permissions to each connectedness such as read from one file and write to a unlike folder.

Annotation: The unique characterization follows the protocol: 'gs+unique_label://restricted_path'

Setting Path Restrictions for Google Cloud Storage Folder and File Access

The default path restrictions are fix to allow admission to all paths in your Google Cloud Storage account, however you lot may want to restrict your algorithm's access to specific folders or files:

• Access to a single file: 'Algorithmia/team.jpg'
• Access to everything in a specific folder: 'Algorithmia/*'

NOTE: 'Algorithmia*' might match more than you'd similar, so if you want to match a directory exactly end with a '/'.

Here we are setting our path restrictions to everything in the Google Deject Storage 'Algorithmia':

Setting Read and Write Admission

The default admission for your data source is set up to read only, just you can change this to read and write admission by checking the 'Write Access' box.

NOTE: Write admission also means you can delete anything in the path y'all've specified in the previous step so be careful that you desire read-write-delete access to the path y'all set in 'Path Restriction'. Also, if your data source has Read/Write privileges, then an algorithm that y'all call too has Read/Write privileges to your data source.

Accessing your Data

Accessing your data via the Algorithmia Information API is easy. Whether you lot're writing your algorithm in Rust, Blood-red, Python, Scala, Java or JavaScript just import your information with a couple lines of code. With your data connection now configured you lot can read and write data to and from it via Algorithmia's Data API by specifying the protocol and label every bit your path to your data:

• customer = Algorithmia.customer('YOUR_API_KEY')
• client.file('gs+unique_label://container_name/my_file.csv').getFile().name

For example, to retrieve and print a file's contents in Python:

                          import              Algorithmia              import              csv              client              =              Algorithmia              .              client              (              'your_api_key'              )              def              google_cloud_data              ():              # Go file from Google Deject Storage data source                            data_file              =              client              .              file              (              'gs+unique_label://Algorithmia/test_data.csv'              ).              become              ()              # Pass in file and pass in args required from the algorithm FpGrowth                            input              =              [              data_file              ,              5              ,              2              ]              algo              =              client              .              algo              (              'paranoia/FpGrowth/0.two.0'              )              return              algo              .              pipage              (              input              )              google_cloud_data              ()                      

The above examples piece of work when accessing data from a local script or app code. If you're writing an algorithm and accessing a information source from inside the algorithm, create the client without an API Key parameter: customer = Algorithmia.customer()

If you're calling an algorithm that takes a file or directory as input from the Data API, yous can also provide it a file or directory from one of your data sources:

                          algo              .              pipeJson              ({              'inputFile'              :              'gs+unique_label://Algorithmia/test_data.csv'              })                      

NOTE: If you telephone call an algorithm information technology can but access your own data sources. This means that information technology is NOT possible for an algorithm to read information from your Google Cloud Storage and write that data to an account controlled by an another algorithm author. Algorithms do Non take straight access to whatsoever credentials associated with your data sources, and tin can simply access data from a data source using the Algorithmia API.

Data Source Routes and Information API Routes

In one case a data source connection has been created and configured, all of the Algorithmia customer lawmaking for interacting with the Data API for file or directory cosmos, deletion and listing will office identically with a data source road and a data API road except for:

• We practise non support generic ACLs for data sources and the only mode to update permissions for a data source is through the information portal where you created your data source connexion.

If you're implementing a new client or using cURL it is preferred to use the following URL construction:

• '/v1/connector/protocol+label/path':
  • '/v1/connector/gs+unique_label/foo/bar.txt'

Algorithm support

We have tested to ensure that information source paths office in all of our Algorithmia clients, however:

• Python support was added in version 1.0.four
• NodeJS back up was added in version 0.3.5 This means that algorithms in Python or JavaScript which were final compiled prior to v/27/2016 might non have the about recent versions of these dependencies, and nosotros can't guarantee this new functionality will work on algorithms older than that. A simple recompilation of the algorithm will enable support without whatsoever lawmaking changes needed.

If you lot have any questions about Algorithmia please go far touch!

campbellformiscrad.blogspot.com

Source: https://algorithmia.com/developers/data/googlecloudstorage

Share this post