Brightpearl API Part IV: Assessment and Plan

Part IV

I’ve been working through a process using the REST web API provided by the Brightpearl Inventory and Customer Relationship Management System. I think we’re about one-third through the process, and I wanted to take stock and plan what the next steps are. I’m mindful of the answer to the question of “how do you eat an elephant?” which is “one bite at a time”.

  • Complete the queries
  • Parse the JSON responses 
  • Convert the results to a .DBF file. 
Since I’m confident about sending queries to the system, I think the next piece to work on is how to parse the JSON output, so that the resulting string of fields and data can be inserted into a database program.  Ultimately, we’re looking to transform the JSON output shown in Part I into a series of fields and data. 
order ID Integer
orderTypeID Integer
contactID Integer
orderStatusID Integer
orderStockStatusID Integer
createdOn Timestamp
createByID Integer
orderPaymentStatusID Integer

Brightpearl API Part III: Working with Resources

This is the third posting regarding the Shopify / Brightpearl API
Part I is located here.
Part II is located here.

The Brightpearl API describes resources that can be queried using an HTTP GET request. Resources look suspiciously like tables in a database.  The Brightpearl resources include: 

Orders
Contact
Return contact information for a single contract with the ID of #200 
GET https://ws-use.brightpearl.com/2.0.0/myaccountid/contact-service/contact/4 
Postal-Address
GET https://ws-use.brightpearl.com/2.0.0/myaccountid/contact-service/postal-address/107
Recall that each of the GET calls above require a valid authorization code to be included in the call as a header 
brightpearl-auth: xxxxxx-xxxxxxx-xxxxxxxx-xxxxxxx-xxxxxxxxxx
Also, that you’d replace myaccountid with the name of your Brightpearl account. 
Since the order returns only the contact ID,  we need to go to contacts to get the person’s name. 
Since the contact only returns the person’s first and last name, we need to go to the address object to get the address. 
So one way that this might work….    
Dimension an array arWorldShip[10], which will hold the following pieces of information: 
order number, contact id, firstName, lastName, address id, address1, address2, address3, address4, postal code, 
Query to get the orders for the day, and then loop through each order 
For each shop order:  
     Store the order number in an arWolrdShip[1]  and the contact ID in arWorldShip[2]
     Query for the customer – to get the first and last name, and address ID
     Store these in the array 
     Query for the address – to get the address information.  (oh and check…for shipping/vs. postal address). 
     Store these in the array 
Next shop order 
Write out the array variables as a record; converting to .DBF.  

Brightpearl API Part II What is JSON?

Part II of a series about working with web APIs. 
 
Part I is located here
 
JSON (JavaScript Object Notation) is described at http://www.json.org/. It is a simple data interchange language which depends on two fundamental data structures:
Object: An unordered set of name / value pairs
Objects are delineated by curly brackets. Each name /value pair is separated by a colon, and pairs are separated by a comma.
Array: An ordered collection of  of values
Values in an array can be almost any data type, including string, number, array, true, false, null, or object.
Arrays are delineated by brackets. Array elements are separated by a comma.
So,  when we’re passing in the authorization credentials for our Brightpearl API, we’re passing in a nested set of name/value pairs as a single object:
{
    apiAccountCredentials:{
        emailAddress:”myname@mydomain.com”,
        password:”mypassword”
    }
}
Brightpearl returns query results as JSON as well. The first part of a the result is the description of the data….the second part is the actual data. (See Part 1 of this series for the full example).
The JSON.org page has links to JSON libraries for a plethora of programming languages and databases. While not listed on the page there is also a JSON module for FileMaker.  For mySQL there is a discussion of a couple ways of formatting the output of a mySQL query into JSON using Ruby, PHP, or using PHPmyAdmin.

Web Services, REST, Shopify and Brightpearl Part I

Part I.

Background:

I am currently working on a project which involves a Shopify online web store, and the Brightpearl Inventory and CRM system. Both of these cloud-based systems have an Application Programmer’s Interface, (API) which provide a programmatic way to query and manipulate the data that has been entered via the normal web interface. They use these APIs to talk to each other and make them available to programmers who want to create custom functionality or plugins for the systems. Communication with these APIs can be done using a REST compatible client written in PHP, Python, Ruby on Rails, or a host of 3rd-generation languages like C# and Visual Basic.
REST stands for Representational State Transfer. This is the most recent flavor of network programming, similar to SOAP, XML, and XML-RPC, and even good old remote procedure calls.

Use-Case:

I’m looking into a way to extract data from the Brightpearl inventory system; I want to query for each day’s purchases and extract the order number, customer name and shipping information. I want to take this information and format it as an .DBF file for use by the UPS WorldShip program. Note that in this example, I’m interested in being a client of an existing web service, and, for the moment I really just need to query the service for existing data, I don’t need to add or delete records on the server.
To start this odyssey, I’m using my Windows workstation. I’m thinking eventually if I need to have a web server for testing (to run PHP or RAILS for example), that I’ll spin that up as a virtual machine using VirtualBox on Windows with Ubuntu Server as my guest OS with a mySQL backend.
The Brighpearl documentation suggests several tools that can be used to send requests to the API. Perverse as it sounds. I found it was helpful to install no less than three add-ons for FoxFire and Chrome to send the API requests, which enabled learning the mechanics of the process a little easier.
For Chrome:
For FireFox:
Each of these three add-ons allow you to send requests to a web server. Each is slightly different. The Chrome add-on includes a parser for JSON data, which is really helpful when you are working with JSON…which is the case with Brightpearl.
Brightpearl also suggests a book from O’Reilly called RESTful Web Services by Leonard Richardson and Sam Ruby. The book was published in 2007, so although it has some useful information, it is somewhat dated. There is nothing about oAuth in it for example.
 
To get started with the Brightpearl API, you have to make sure that your user account is authorized to work with the API. This is done by accessing the “Staff” under Setup, and making sure that there is a green checkmark next to the user’s name in the API access column. 
Get an Authorization Token
Brightpearl requires that you obtain an authorization token prior to accessing any other requests.  The request for the authorization token takes the form of  a POST request  which includes your user name and password in the request payload. The URI of the payload includes two variables,  your brightpearl server location, the name of your BrightPearl account and a Content-Type of text/xml
Content-Type: text/xml
where
use=”US East”
“microdesign”, is the name of your Brightpearl account id
The user name and password are passed as JSON name pairs to the apiAccountCredentials variable:
{

    apiAccountCredentials:{
        emailAddress:”myname@mydomain.com”,
        password:”mypassword”
    }

}
Note that the double quotes enclosing the eMail address and password are also present.
So, if you look at the raw request that is sent, the full request looks like this:
POST https://ws-use.brightpearl.com/microdesign/authorise
Content-Type: text/xml
{
apiAccountCredentials:{
emailAddress:”myname@mydomain.com”,
        password:”mypassword”    }
}
If the request is successful, you’ll receive a hexedicimal number back which is your authorization token.
{“response”:” xxxxxx-xxxxx-xxxxx-xxxxx-xxxx-xxxxxxxxxx”}
Once you have the authorization token, it is used in subsequent requests as a substitute for your user name and password. The token expires after about 30 minutes of inactivity…so you’ll have to issue another authorization request and obtain a new token after that time. 
Once you have gotten the authorization token, you can start making requests. The basic request is a “resource search” which is a query of the Brightpearl data. Resource searches are issued with GET requests, and must include the API version number. The authorization code is sent as a header along with the request. 
 
As a reminder, the authorization request is a POST, and the resource query is a GET.
(More on resource searches in Brightpearl).
GET https://ws-use.brightpearl.com/2.0.0/microdesign/warehouse-service/goods-note/goods-out-search
brightpearl-auth: xxxxxx-xxxxx-xxxxx-xxxxx-xxxx-xxxxxxxxxx
This request returns a list of the current goods-out notes (Brightpearl’s nomenclature for a packing slip or pick-list).
Example with results: 
The folllowing GET request shows the current orders.
brightpearl-auth: xxxxxx-xxxxx-xxxxx-xxxxx-xxxx-xxxxxxxxxx
This returns a list of current orders, in JSON format. The format shows the structure of the data first, and then the actual records.  Note that there are only three orders!
{“response”:{“metaData”:{“resultsAvailable”:3,”resultsReturned”:3,”firstResult”:1,”lastResult”:3,”columns”:[{“name”:”orderId”,”sortable”:true,”filterable”:true,”reportDataType”:”IDSET”,”required”:false},{“name”:”orderTypeId”,”sortable”:true,”filterable”:true,”reportDataType”:”INTEGER”,”referenceData”:[“orderTypeNames”],”required”:false},{“name”:”contactId”,”sortable”:true,”filterable”:true,”reportDataType”:”INTEGER”,”required”:false},{“name”:”orderStatusId”,”sortable”:true,”filterable”:true,”reportDataType”:”INTEGER”,”referenceData”:[“orderStatusNames”],”required”:false},{“name”:”orderStockStatusId”,”sortable”:true,”filterable”:true,”reportDataType”:”INTEGER”,”referenceData”:[“orderStockStatusNames”],”required”:false},{“name”:”createdOn”,”sortable”:true,”filterable”:true,”reportDataType”:”PERIOD”,”required”:false},{“name”:”createdById”,”sortable”:true,”filterable”:true,”reportDataType”:”INTEGER”,”required”:false},{“name”:”customerRef”,”sortable”:true,”filterable”:true,”reportDataType”:”STRING”,”required”:false},{“name”:”orderPaymentStatusId”,”sortable”:true,”filterable”:true,”reportDataType”:”INTEGER”,”referenceData”:[“orderPaymentStatusNames”],”required”:false}],”sorting”:[{“filterable”:{“name”:”orderId”,”sortable”:true,”filterable”:true,”reportDataType”:”IDSET”,”required”:false},”direction”:”ASC”}]},”results”:[[1,1,207,4,3,”2014-09-18T14:15:50.000-04:00″,4,”#1014″,2],[2,1,207,1,3,”2014-09-29T13:20:52.000-04:00″,4,”#1015″,2],[3,1,207,1,3,”2014-09-29T13:25:39.000-04:00″,4,”#1016″,2]]},”reference”:{“orderTypeNames”:{“1″:”SALES_ORDER”},”orderPaymentStatusNames”:{“2″:”PARTIALLY_PAID”},”orderStatusNames”:{“1″:”Draft / Quote”,”4″:”Invoiced”},”orderStockStatusNames”:{“3″:”All fulfilled”}}}
If you use the “Advanced REST Client Application For Chrome, it will decode the above so that it is readable:
{
response:

{
metaData:

{
resultsAvailable3
resultsReturned3
firstResult1
lastResult3
columns:

[

9]

0:  

{
name: “orderId
sortabletrue
filterabletrue
reportDataType: “IDSET
requiredfalse
}
1:  

{
name: “orderTypeId
sortabletrue
filterabletrue
reportDataType: “INTEGER
referenceData:

[

1]

0:  orderTypeNames
requiredfalse
}
2:  

{
name: “contactId
sortabletrue
filterabletrue
reportDataType: “INTEGER
requiredfalse
}
3:  

{
name: “orderStatusId
sortabletrue
filterabletrue
reportDataType: “INTEGER
referenceData:

[

1]

0:  orderStatusNames
requiredfalse
}
4:  

{
name: “orderStockStatusId
sortabletrue
filterabletrue
reportDataType: “INTEGER
referenceData:

[

1]

0:  orderStockStatusNames
requiredfalse
}
5:  

{
name: “createdOn
sortabletrue
filterabletrue
reportDataType: “PERIOD
requiredfalse
}
6:  

{
name: “createdById
sortabletrue
filterabletrue
reportDataType: “INTEGER
requiredfalse
}
7:  

{
name: “customerRef
sortabletrue
filterabletrue
reportDataType: “STRING
requiredfalse
}
8:  

{
name: “orderPaymentStatusId
sortabletrue
filterabletrue
reportDataType: “INTEGER
referenceData:

[

1]

0:  orderPaymentStatusNames
requiredfalse
}
sorting:

[

1]

0:  

{
filterable:

{
name: “orderId
sortabletrue
filterabletrue
reportDataType: “IDSET
requiredfalse
}
direction: “ASC
}
}
results:

[

3]

0:  

[

9]

0:  1
1:  1
2:  207
3:  4
4:  3
5:  2014-09-18T14:15:50.000-04:00
6:  4
7:  #1014
8:  2
1:  

[

9]

0:  2
1:  1
2:  207
3:  1
4:  3
5:  2014-09-29T13:20:52.000-04:00
6:  4
7:  #1015
8:  2
2:  

[

9]

0:  3
1:  1
2:  207
3:  1
4:  3
5:  2014-09-29T13:25:39.000-04:00
6:  4
7:  #1016
8:  2
}
reference:

{
orderTypeNames:

{
1: “SALES_ORDER
}
orderPaymentStatusNames:

{
2: “PARTIALLY_PAID
}
orderStatusNames:

{
1: “Draft / Quote
4: “Invoiced
}
orderStockStatusNames:

{
3: “All fulfilled
}
}
}

A Database for Grant Research

I put together a grants database screen (click to view full size) to consolidate information for funding sources, and to track dates and interactions.

It is definitely an evolving project, but contains the basic information need to contact the funder, the deadline dates involved, the funder’s areas of interest, and the typical range of a grant award.

So far, I’ve been concentrating on foundation funding. Many foundations typically ask for a letter of interest before you put together a full proposal. So, I’ve included multiple date fields, a deadline for a letter of interest, a deadline for a full proposal, and a date when they announce their award.

Originally I thought that this database would be mostly for research, but after working with the online grants database, Grantstation, I think I will reserve this database for funders that I really expect to submit to. Some ideas for future enhancements include:

  • Links to standard “boilerplate” paragraphs that are used in an application. 
  • Links to edit the proposal or letter directly in Word. 
  • Links to the PDFs of the proposal. 
  • Reports that create a grants calendar. 
Before anyone comments that “you should really use X software” for this purpose, I just want to say that I’ve used several in the past, including DonorPerfect and Blackbaud, and evaluated many others. Right now, I’m in the process of rethinking my entire workflow automation from the ground up, and this very lightweight approach is just what I’m looking for. Plus its in FileMaker, so I can run it on my Windows machines at work, or my Macs at home.     

Odds and Sods: Grantstation Membership and Training Opportunities

Grantwriting

Grantstation is a subscription-based database of grant opportunities. You can purchase a l year Grantstation Membership for $99.00 per year if you are a member of TechSoup, on September 23 and 24.

The free Grantstation 1-hour orientation webinar is well worth the time. I learned, for example that their federal grants database is a front-end using data from Grants.gov.    

Tech Soup is also sponsoring a free webinar by the CEO of Grantstation Cynthia Adams on September 18th.

Funding Rural America :
When has it ever been easy to secure funding for nonprofits and libraries in rural communities? Is there a way to level the playing field so organizations in small towns, counties, or boroughs without large metropolitan areas can compete for both government and private sector grants? What are the other options for financially supporting a rural project? Are collaborative efforts worth the effort? And, of course the biggest question: who is funding rural America?

These, and other relevant questions, along with current trends affecting rural funding will be addressed in this free, two-hour webinar presented by Cynthia Adams, CEO of GrantStation.

Finally, it is rare that we find a Grantmsanship Seminar in our neck of the woods (northern New England). But  there will be a five-day Grantmanship training seminar located in Barre VT. December 8-12, 2014

Hardware and Software

Interconnection.org is  another possible source for refurbished computers, (or a place to donate your old gear). They are one of the refurbishers for the Techsoup Refurbished Computer Initiative.

Techsoup has wonderful deals for software. A one-year subscription to FileMaker server is $209, and a subscription for FileMaker Pro (the desktop version that runs on a Mac or PC desktop) is $65.00. Visio Professional is available for $29.00.

Miscellaneous

Here is an interesting academic discussion of luck, and how to have more of it. Thanks to Jeff Duntemann for the link.

Rebootolator: Execute a Remote Linux Shell Script from Windows

Ok,  so, your mission, should you decide to accept it, is to restart mySQL and Apache on a remote server. This restarts a balky web site hosted by Apache, and also restarts a mySQL server which is used for a back-end for Drupal.

You want to execute this from your Windows computer.

The target computer runs CentOS 5.6 This is an (ancient) Red Hat Linux derivative, running (ancient) mySQL and Apache.

I ended up using PLink called from a Windows .CMD file to execute a bash shell script.  The shell script looks like this:

Rebootolator

#!/bin/bash -p
# Rebootolator – Reboots Apache and mySQL on a target Server
# LK Microdesign June 25, 2014
TERM=”xterm”
export TERM
clear
echo
echo ‘Rebooting Apache and mySQL on myServer’
echo ‘———————————–‘
echo ‘Restarting mySQL’
/etc/init.d/httpd restart
echo ”
echo ‘Restarting the Apache web server.’
/etc/init.d/mysqld restart
echo ‘Reboot procedure completed’


Note this script is not stored on the target server, but simply put in the same folder as the windows cmd file on my windows box.  

Now for the Windows command file: 

Reboot.CMD

:: Batch file to restart services on myServer
:: Restarts mySQL and httpd 
:: Uses the Rebootolator shell script
:: LK/Microdesign August 12, 2014  
@echo off
cls
echo. 
echo.
plink -ssh username@192.168.xxx.xxx -m rebootolator.sh -pw mypass
echo.
echo.

pause >nul | echo Press any key to exit. 

So, lets deconstruct the Windows Reboot.CMD file.
The first four lines are comment lines. Turns out, you can use two colons to preface a comment in Windows, (who knew?) instead of REM.
Line 5 turns off output to the screen.
Line 6 clears the screen.
Line 7 and 8 put in blank lines.
All the work happens on line 9, using the PLINK command. PLINK is the command line version of PUTTY, a free open source terminal program for Windows workstations. Both PLINK and PUTTY are pretty wonderful and highly recommended if you need to access Linux machines from Windows.
-ssh means “use the secure socket layer protocol to log into this machine”
username@192.168.xxx.xxx is a administrator’s account on the target machine,  probably the root account.
-m rebootolator.sh is the name of the shell script (above) that needs to run on the target machine.
-pw mypass is the password for the account used to log into the machine.

Deconstructing the Rebootolator.sh script:
#!/bin/bash -p  just means this is a BASH script
The two commands that actually restart the mySQL server, and the Apache server are: 
/etc/init.d/httpd restart
/etc/init.d/mysqld restart

The rest, (the echo commands) write out what  is happening at the command line. The Term command is my attempt to avoid a harmless error message that occurs when the script starts to execute.

Since I didn’t realize I could host the Rebootolator.sh script in my Windows folder, I originally though I’d have to log into one Linux box, and then execute the script on the target box.  Turned out the whole thing was simpler using PLINK, which is the equivalent of SSH and SSHPASS programs used to access remote machines from the Linux command line.

Tech Friday: FileMaker Resources

Random FileMaker-related resources: 

1. You can create an alternative icon for FileMaker 13. When you have multiple FileMaker versions installed, it sometimes is difficult to distinguish between them. FM12 and FM13 have virtually identical icons with the same color schemes. Here are alternative icons that can be installed, for both Win and Mac from HomeBase Software. HomeBase has a ton of of technical information on their web site.

2. In support of a project to integrate SmartyStreets with FileMaker, I’ve been doing some additional research on JSON, (Javascript Object Notation), which is a simplified version of XML.

3. Coding Standards for FileMaker

4. Modular FileMaker: is shared library of FileMaker functions. There are huge community-developed libraries for other languages such as PHP and Python. These folks are attempting a similar idea for FM. I’ve downloaded their JSON module, and am experimenting with it. Other examples include a nifty SQL query generator, and and another interface to Mailchimp.

Custom Functions are a way of adding small chunks of user-defined code that can be called within a FileMaker script. Brian Dunning is the guru here and curates the largest library of custom FM functions on the web. He also has sample data sets available for the U.S., Austria, Canada, and the UK. Five hundred records are free, and a million records are available for ten bucks.

5. If you need a 100,000 records or so, you could also download the database of public and private schools available at the National Center for Education Statistics. This includes demographic data as well as mailing and location addresses for schools. The data is fun to play around with. You can give yourself some sample exercises in FileMaker. For example:

6. What percentage of public school students are eligible for free or reduced lunch in your state? It is 38% in mine. Poking around in some other states, it looks like that isn’t unusual; in many states it is 40-60% or more. The lunch program is often considered a proxy for the family poverty rate. But maybe that’s another discussion.

 

Sync Google Apps with Microsoft OutLook 2013

I installed Microsoft Outlook 2013 the other day, along with the rest of Office 2013. I’m only about two years late, right?  Office 2013 offers little over Office 2010, but there is a somewhat cleaner design (OSX 10.7 anyone?) and really nice animations when windows open and close. Forgive me for saying this, but the interface is more Mac-like, even though it came out two years before the recent updates to OSX and IOS 7.

I originally had OutLook eMail synced to my Google Apps email. Then I wanted to also use the Outlook calendar and have changes made there reflected in my Google Apps calendar which is the group calendar that we use in the office. A quick search found the Google Outlook synchronizer, an application that sits outside of Outlook and runs interference between Outlook and Google apps.

The nice thing about this is that that it doesn’t require you to put in all kinds of IMAP information into your Outlook profile…you simply give it your Google Apps eMail address and off it goes and downloads everything from Google Apps, creates a new profile within OutLook, and then uploads everything into mailboxes in Outlook (see screenshot). If it works, it will great. I includes your contact lists, shared calendars, co-workers calendars, notes, and tasks.

The poor mans’s Exchange Server? 

If you use Google Apps in the office, and have Windows computers, you could have this running on each machine, and essentially use Google Apps as a back-end Exchange Server.

SmartyStreets – Validate Mailing Addresses

Some months ago I submitted a mailing list file to our mailer (we use Quad), and I was somewhat taken aback to get their report that over 5% of the addresses that we sent to them were invalid or incorrect.. I was thinking this wasn’t too bad a figure, but they informed me that “people who knew what they were doing” would achieve valid rating of close to 100%. My thought was to do some “pre-validation” before sending the mail file to Quad. Turns out this can get expensive. Then I found SmartyStreets.

SmartyStreets is a web-based address validator for U.S. addresses. Using the US Postal Service official address database, SmartyStreets will validate any address that you send to it with varying amounts of correction.  Addresses are classified in a number of different ways, including:

  • Nomatch             The address is invalid.
  • Mailable              The address is valid and can receive mail. 
  • Mailable-Vacant:  The address is valid but vacant
  • Match-Inactive.   The address is valid but inactive. 

SmartyStreets has a one-off web interface which allows you to validate addresses one at a time on the fly. http://www.smartystreets.com. It will supply a nine digit zip code for valid street address, and it will suggest addresses nearby if you submit an invalid address. In short, if SmartyStreets returns an address, it will most probably be mailable. That’s what I’m hoping, anyway. I just sent a new file to Quad, and we’ll see what the accuracy is.

Click to Enlarge.
The web-based interface returns, not just a corrected address but additional information such as lattitude and longitude of the address, whether it is commercial or residential,and the address’s time zone and congressional district. 
Smartystreets will process lists of addresses. You can simply paste in an ASCII comma-delimited list or an Excel file containing an address list, and SmartyStreets will return the list with in a mail/no-mail format, or with about 30 fields of additional information. 
SmartyStreets frequently donates their services to churches, schools, libraries, and many non-profit organizations. They have extensive documentation that explains how the validation process works, and what the results of your file can contain. And they have an API (an application programmer’s interface) which will return results in XML or JSON format.  
A great addition to the mailing toolbox.