Tag Archives: Linux

Moving to the Cloud – with Box Part 1

We’re moving to the cloud with cloud storage for working files. Old news of course,  haven’t we had cloud storage for years already?  Of course… let me count the ways:

  • Adobe Creative Cloud Libraries
  • Apple iDrive
  • Microsoft OneDrive
  • Microsoft Azure
  • Amazon Web Services
  • Google Drive
  • DropBox
  • Box

A quick Google search also shows up some open source solutions that you could install on your Linux server.  But today, we’ll take a look at Box.

The wonderful TechSoup has an offer for Box at the “starter” level  for 10 users for $84.00/year. This is just about right for our workgroup; we currently have 8 full and part-timers on our team, which leaves 2 additional slots available for what we hope we have for growth in the next year. While we do have an office, we are a distributed group. Each full-timer spends a minimum of one day per week outside the office, and our part time employees either work from home, or come in during only part of their week.

What we’re trying to replace here is is an in-office rack-mounted physical server. (remember those?) which sits in a corner of the office roaring away, much as it has for at least ten years. This is a Linux server running the Samba file-management system which is solid and reliable, but a pain to manage. We typically map to drive letters on each person’s workstation:

Drive F: – This letter is mapped to the user’s personal folder on the server. So, my case, my F: drive is mapped to //server/home/larry

Drive U: – This letter is mapped to our “Main” shared folder, under which there are about a dozen departmental or functional sub-folders including Admin, Creative, Editorial, Grants,  etc.

On Linux if you know how Samba works; (and a GUI interface is really helpful…) you can restrict each of the folders to groups of appropriate users. So, for example, you can restrict the HR folder to your bookkeeper,  HR manager and your E.D. There is an additional complication with Samba in that you have to maintain a parallel set of Linux logins and home directories for each Samba user.  Box provides the ability to maintain a similar set of permissions and file restrictions within a web interface. Even thought the “starter” version isn’t as versatile as their full version it still allows you assign individual users as “collaborators” for individual folders.

Other user requirements:

  • Cross-platform availability,  Mac, Windows, iOS, Android
  • Native applications for each platform.
  • Available from anywhere with an internet connection
  • Ability to sync between the cloud and the device.
  • Butt-simple interface that passes the five minute test.

Next time we’ll get into more detail about Box.

 

 

 

 

Troubleshoot Apache with mod_status

We are having a bit of a contretemps with one our of Linode blog hosts. This host is running Ubuntu Linux 12X  LTS and WordPress, and seems to be having fits of high CPU utilization. One way to see into the web hosting process is to examine the Apache mod_status page.

Mod_status is an Apache module which prints statistics about the Apache application. By default the page will be accessible using the the suffix of /server-status on your root web site URL. http://mysite.com/server-status. However, by default this is restricted to a request from a browser at the 127.0.0.1 address.

If you log into the console, you can attempt to see the status by running the following:

apachectl status

or if you aren’t logged in as root,

sudo apachectl status

This returns the status page.

Screenshot_071516_011101_PM

If you are attempting to access this from a browser on another workstation via http://mysite.com/server-status, you get a 403 message:

Screenshot_071516_110946_AM

The first thing is to find where this restriction is located. It is going to be in one of the Apache configuration files. These are located within /etc/apache2. If the status module is enabled, there will be a status.conf file that can be edited.

sudo nano  /etc/apache2/mods-enabled/status.conf

Screenshot_071516_112041_AM

Edit the lines after the <Location /server-status> line, per the instructions. After editing mine looked like:

<Location /server-status>
   SetHandler server-status
   #Require local 
   Require ip 192.168.xxx.0/24
</Location>

where “xxx” is your local subnet.  In the above example, I’m accessing the server from inside the firewall from a workstation located on the same subnet as the Apache server. (In reality, I’m actually accessing a server running within a VirtualBox virtual machine, located on my Windows machine. )

Once these changes are made you have to restart Apache.

service apache2 restart

 

If this is successful,  you get a page similar to the character-based page, but nicely formatted in html. Since the data is similar, however, if there is any issue trying to get at the statistics, probably the character-based method at the console is the first thing to try.

Screenshot_071516_011525_PM

Ah…but what if you get a page, but it isn’t a status page?  This is the problem we have with our WordPress site, and it has to do with which page is served as the default. http://myblog.com/server-status  returns the default page from the blog rather than the server-status page.  Stay tuned for that fix.

Hosting with Linode

Let us sing the praises of Linode, a provider of cloud-hosting for Linux-based applications and web sites. At work we’ve had two of these for a year, and they just work, exactly if the servers were sitting in the next room, except that they provide a slick management interface, and you can mix and match among different disk sizes, RAM, processors, redundancy, and various flavors of Linux. Access is via their web site, or via a secure terminal session.

Here’s a shot of the backup screen, (click to enlarge).

 The Linode-provided backup is great for basic backup. It makes backups on a weekly and daily basis. It also has a provision for a snapshot, which is a great way to back up the server before doing upgrades. Full details are on their web site, including the costs, which are a few dollars per month depending on the size of the Linode.

There are some limitations…for example with MySQL database transactions. Linode recommends that you perform a data dump of your MySQL data so that the dump files get backed up.

PowerShell FTP Follow-up

This script is an attempt to automate a lengthy error-prone copying and configuration process that we do each week. If we do the process manually it can take anywhere from five to twenty minutes, and it tends to have various points of failure.  The piece below is actually just one portion of the process. The steps include: 

1. Get the name of a new folder to be created on the server
2. Get the name of the file to be copied into the new folder
3. Using the two new names, build a text file which contains commands that will be fed into PSFTP
4. Call PSFTP and run the commands in the text file.

This  PowerShell script uses Putty FTP to log into an FTP server, create a new folder, and copy a file to that folder from the local host. Note the the steps for making the folder and copying the file are contained in a Putty script called gwkprocess.scr. This secondary script is is used as input to the Putty program after Putty makes the connection.  Those steps are typical FTP steps: 

CD / topdirectory
MKDIR  /new directory
CD /newdirectory 
PUT myfile.png  


<# Powershell Scripted FTP
LK 10.30.2014 
Send a file to the eMail server via FTP.
Uses the Putty Secure FTP program PSFTP
#>


# $FTPFolder=’/home/web/html/store/images/fy2014/Kids-Shop’
# Note that the login credentials are in clear text! 

# Enter the new folder name here. 
$NewFolder = “20141101ks”

# The Picture file to be copied is located in 
# C:UsersLarryPowershell 
# and should be named, with the usual naming convention
$PicFile = “20141101ks-image.png”

#Note line wraps. 

#Build the Putty Script file 
“cd /home/web/html/store/images/fy2014/Kids-Shop”| Out-File -FilePath C:UsersLarryPowershellgwkprocess.scr -Encoding ascii

“mkdir $Newfolder” | Out-File -FilePath  C:UsersLarryPowershellgwkprocess.scr  -Encoding ascii -Append


“put $PicFile” | Out-File -FilePath  C:UsersLarryPowershellgwkprocess.scr  -Encoding ascii -Append

“ls” | Out-File -FilePath C:UsersLarryPowershellgwkprocess.scr  -Encoding ascii -Append


# Call the putty program 
.psftp myuser@192.168.214.103 -P 22 -pw mypassword -v -2 -b gwkprocess.scr

This starts PSFTP in the Powershell window, makes the connection and then executes the gwkprocess.scr  steps. It then closes the connection. If there is a problem, PSFTP will print a failure message, but clearly there is room for more error checking on the front end. 

The presumption is that the secondary script gets rebuilt with new file and folder names each time the script is run. Obviously, there are some refinements to be included, like  interactive data entry of the file and folder names.

Powershell: Scripting FTP

Having spent some hours figuring out how to script an FTP transfer, I thought I’d describe my kludge. Maybe someone can suggest a more elegant way.  I’m trying to I’m connect to a FTP server on my Linux box to upload a file.
There are at least three approaches that can be taken:
1. Directly interact with .NET objects
2. Import a Powershell module for FTP
3. Use Powershell to manipulate a command line FTP program, such as the Putty Secure FTP program PSFTP.
I started with the second option recommended on TechNet. Looks great, and I thought that it was semi-official (being from Technet). I was unable to get a connection and I think it may be related to the fact that module apparently doesn’t support SFTP version 2.  There are a couple other quirks with the module… including the fact that the user name and password are passed to the command line as an object.  
By the way, both option 2 and 3 have the same name, PSFTP. 
Option 2 = Powershell FTP 
Option 3 = Putty Secure FTP 
So, I’m on to option 3.  This looks a little more promising.  One gotcha, however, is that calling Putty PSFTP from the Powershell ise, makes the connection but doesn’t return to show the PSFTP prompt. Here’s the command (so far) 
PS> .psftp myaccount@192.168.224.184 -p 22 -pw mypassword -v -2
This command shows that:
The psftp program is located in the current directory.
myaccount@192.168.224.184 – is the login account used for logging into the target machine
192.168.224.185 – is the IP address of the target machine
-p 22 – is port 22, used for Secure FTP
-pw is the password
-v is verbose (upon execution it returns all the steps of the login
-2 is SSL version 2.
Running this from the command line in the ISE gives the following:
  .psftp : Looking up host “192.168.214.184”
At line:1 char:1
+ .psftp myaccount@192.168.224.184 -P 22 -pw mypassword -v -2
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Looking up host “192.168.224.184”:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
Connecting to 192.168.224.184 port 22
We claim version: SSH-2.0-PuTTY_Release_0.63
Server version: SSH-2.0-OpenSSH_3.8.1p1
Using SSH protocol version 2
Doing Diffie-Hellman group exchange
Doing Diffie-Hellman key exchange with hash SHA-1
Host key fingerprint is:
ssh-rsa 1024 ce:ec:0d:c2:90:ab:5e:87:12:bf:ba:f9:78:77:89:fb
Initialised AES-256 SDCTR client->server encryption
Initialised HMAC-SHA1 client->server MAC algorithm
Initialised AES-256 SDCTR server->client encryption
Initialised HMAC-SHA1 server->client MAC algorithm
Using username “myaccount”.
Attempting keyboard-interactive authentication
Access granted
Opening session as main channel
Opened main channel
Started a shell/command
Connected to 192.168.214.103
Remote working directory is /ftproot
PS >
If you run this from the ise, it returns the PS command as shown.  
If you run this from a regular powershell command session, it will keep you in the PSFTP session, and you can run use the usual FTP commands….like CD, etc. 
In either case, the way Putty FTP can execute scripted commands is that the script has to live in a separate batch file that is called from the command line.  There doesn’t appear to be a  way to pass commands from Powershell to a running Putty SFTP session.  (hmm.. really?) 
Additioanal points: 
1. I might be able to configure the FTP server running on the Linux box to accept the connection from the PowerShell FTP module. I haven’t investigated that possibility but presumably it would be less secure in terms of encryption.

2. The script above has the name and password in clear text.  Not a best practice. 
3. Various versions of FTP are described on this page.  

Web Services, REST, Shopify and Brightpearl Part I

Part I.

Background:

I am currently working on a project which involves a Shopify online web store, and the Brightpearl Inventory and CRM system. Both of these cloud-based systems have an Application Programmer’s Interface, (API) which provide a programmatic way to query and manipulate the data that has been entered via the normal web interface. They use these APIs to talk to each other and make them available to programmers who want to create custom functionality or plugins for the systems. Communication with these APIs can be done using a REST compatible client written in PHP, Python, Ruby on Rails, or a host of 3rd-generation languages like C# and Visual Basic.
REST stands for Representational State Transfer. This is the most recent flavor of network programming, similar to SOAP, XML, and XML-RPC, and even good old remote procedure calls.

Use-Case:

I’m looking into a way to extract data from the Brightpearl inventory system; I want to query for each day’s purchases and extract the order number, customer name and shipping information. I want to take this information and format it as an .DBF file for use by the UPS WorldShip program. Note that in this example, I’m interested in being a client of an existing web service, and, for the moment I really just need to query the service for existing data, I don’t need to add or delete records on the server.
To start this odyssey, I’m using my Windows workstation. I’m thinking eventually if I need to have a web server for testing (to run PHP or RAILS for example), that I’ll spin that up as a virtual machine using VirtualBox on Windows with Ubuntu Server as my guest OS with a mySQL backend.
The Brighpearl documentation suggests several tools that can be used to send requests to the API. Perverse as it sounds. I found it was helpful to install no less than three add-ons for FoxFire and Chrome to send the API requests, which enabled learning the mechanics of the process a little easier.
For Chrome:
For FireFox:
Each of these three add-ons allow you to send requests to a web server. Each is slightly different. The Chrome add-on includes a parser for JSON data, which is really helpful when you are working with JSON…which is the case with Brightpearl.
Brightpearl also suggests a book from O’Reilly called RESTful Web Services by Leonard Richardson and Sam Ruby. The book was published in 2007, so although it has some useful information, it is somewhat dated. There is nothing about oAuth in it for example.
 
To get started with the Brightpearl API, you have to make sure that your user account is authorized to work with the API. This is done by accessing the “Staff” under Setup, and making sure that there is a green checkmark next to the user’s name in the API access column. 
Get an Authorization Token
Brightpearl requires that you obtain an authorization token prior to accessing any other requests.  The request for the authorization token takes the form of  a POST request  which includes your user name and password in the request payload. The URI of the payload includes two variables,  your brightpearl server location, the name of your BrightPearl account and a Content-Type of text/xml
Content-Type: text/xml
where
use=”US East”
“microdesign”, is the name of your Brightpearl account id
The user name and password are passed as JSON name pairs to the apiAccountCredentials variable:
{

    apiAccountCredentials:{
        emailAddress:”myname@mydomain.com”,
        password:”mypassword”
    }

}
Note that the double quotes enclosing the eMail address and password are also present.
So, if you look at the raw request that is sent, the full request looks like this:
POST https://ws-use.brightpearl.com/microdesign/authorise
Content-Type: text/xml
{
apiAccountCredentials:{
emailAddress:”myname@mydomain.com”,
        password:”mypassword”    }
}
If the request is successful, you’ll receive a hexedicimal number back which is your authorization token.
{“response”:” xxxxxx-xxxxx-xxxxx-xxxxx-xxxx-xxxxxxxxxx”}
Once you have the authorization token, it is used in subsequent requests as a substitute for your user name and password. The token expires after about 30 minutes of inactivity…so you’ll have to issue another authorization request and obtain a new token after that time. 
Once you have gotten the authorization token, you can start making requests. The basic request is a “resource search” which is a query of the Brightpearl data. Resource searches are issued with GET requests, and must include the API version number. The authorization code is sent as a header along with the request. 
 
As a reminder, the authorization request is a POST, and the resource query is a GET.
(More on resource searches in Brightpearl).
GET https://ws-use.brightpearl.com/2.0.0/microdesign/warehouse-service/goods-note/goods-out-search
brightpearl-auth: xxxxxx-xxxxx-xxxxx-xxxxx-xxxx-xxxxxxxxxx
This request returns a list of the current goods-out notes (Brightpearl’s nomenclature for a packing slip or pick-list).
Example with results: 
The folllowing GET request shows the current orders.
brightpearl-auth: xxxxxx-xxxxx-xxxxx-xxxxx-xxxx-xxxxxxxxxx
This returns a list of current orders, in JSON format. The format shows the structure of the data first, and then the actual records.  Note that there are only three orders!
{“response”:{“metaData”:{“resultsAvailable”:3,”resultsReturned”:3,”firstResult”:1,”lastResult”:3,”columns”:[{“name”:”orderId”,”sortable”:true,”filterable”:true,”reportDataType”:”IDSET”,”required”:false},{“name”:”orderTypeId”,”sortable”:true,”filterable”:true,”reportDataType”:”INTEGER”,”referenceData”:[“orderTypeNames”],”required”:false},{“name”:”contactId”,”sortable”:true,”filterable”:true,”reportDataType”:”INTEGER”,”required”:false},{“name”:”orderStatusId”,”sortable”:true,”filterable”:true,”reportDataType”:”INTEGER”,”referenceData”:[“orderStatusNames”],”required”:false},{“name”:”orderStockStatusId”,”sortable”:true,”filterable”:true,”reportDataType”:”INTEGER”,”referenceData”:[“orderStockStatusNames”],”required”:false},{“name”:”createdOn”,”sortable”:true,”filterable”:true,”reportDataType”:”PERIOD”,”required”:false},{“name”:”createdById”,”sortable”:true,”filterable”:true,”reportDataType”:”INTEGER”,”required”:false},{“name”:”customerRef”,”sortable”:true,”filterable”:true,”reportDataType”:”STRING”,”required”:false},{“name”:”orderPaymentStatusId”,”sortable”:true,”filterable”:true,”reportDataType”:”INTEGER”,”referenceData”:[“orderPaymentStatusNames”],”required”:false}],”sorting”:[{“filterable”:{“name”:”orderId”,”sortable”:true,”filterable”:true,”reportDataType”:”IDSET”,”required”:false},”direction”:”ASC”}]},”results”:[[1,1,207,4,3,”2014-09-18T14:15:50.000-04:00″,4,”#1014″,2],[2,1,207,1,3,”2014-09-29T13:20:52.000-04:00″,4,”#1015″,2],[3,1,207,1,3,”2014-09-29T13:25:39.000-04:00″,4,”#1016″,2]]},”reference”:{“orderTypeNames”:{“1″:”SALES_ORDER”},”orderPaymentStatusNames”:{“2″:”PARTIALLY_PAID”},”orderStatusNames”:{“1″:”Draft / Quote”,”4″:”Invoiced”},”orderStockStatusNames”:{“3″:”All fulfilled”}}}
If you use the “Advanced REST Client Application For Chrome, it will decode the above so that it is readable:
{
response:

{
metaData:

{
resultsAvailable3
resultsReturned3
firstResult1
lastResult3
columns:

[

9]

0:  

{
name: “orderId
sortabletrue
filterabletrue
reportDataType: “IDSET
requiredfalse
}
1:  

{
name: “orderTypeId
sortabletrue
filterabletrue
reportDataType: “INTEGER
referenceData:

[

1]

0:  orderTypeNames
requiredfalse
}
2:  

{
name: “contactId
sortabletrue
filterabletrue
reportDataType: “INTEGER
requiredfalse
}
3:  

{
name: “orderStatusId
sortabletrue
filterabletrue
reportDataType: “INTEGER
referenceData:

[

1]

0:  orderStatusNames
requiredfalse
}
4:  

{
name: “orderStockStatusId
sortabletrue
filterabletrue
reportDataType: “INTEGER
referenceData:

[

1]

0:  orderStockStatusNames
requiredfalse
}
5:  

{
name: “createdOn
sortabletrue
filterabletrue
reportDataType: “PERIOD
requiredfalse
}
6:  

{
name: “createdById
sortabletrue
filterabletrue
reportDataType: “INTEGER
requiredfalse
}
7:  

{
name: “customerRef
sortabletrue
filterabletrue
reportDataType: “STRING
requiredfalse
}
8:  

{
name: “orderPaymentStatusId
sortabletrue
filterabletrue
reportDataType: “INTEGER
referenceData:

[

1]

0:  orderPaymentStatusNames
requiredfalse
}
sorting:

[

1]

0:  

{
filterable:

{
name: “orderId
sortabletrue
filterabletrue
reportDataType: “IDSET
requiredfalse
}
direction: “ASC
}
}
results:

[

3]

0:  

[

9]

0:  1
1:  1
2:  207
3:  4
4:  3
5:  2014-09-18T14:15:50.000-04:00
6:  4
7:  #1014
8:  2
1:  

[

9]

0:  2
1:  1
2:  207
3:  1
4:  3
5:  2014-09-29T13:20:52.000-04:00
6:  4
7:  #1015
8:  2
2:  

[

9]

0:  3
1:  1
2:  207
3:  1
4:  3
5:  2014-09-29T13:25:39.000-04:00
6:  4
7:  #1016
8:  2
}
reference:

{
orderTypeNames:

{
1: “SALES_ORDER
}
orderPaymentStatusNames:

{
2: “PARTIALLY_PAID
}
orderStatusNames:

{
1: “Draft / Quote
4: “Invoiced
}
orderStockStatusNames:

{
3: “All fulfilled
}
}
}