Scripting 101

["Dave Harper" <dharper@xxxxxxxxxxxxxxxx>]

Scripting 101

This post is in response to a query in another thread regarding a
comment I made about scripting.  I use scripting extensively in my home
automation environment and it allows me to quickly create software to
control the diverse hardware I have installed.

First, a disclaimer; I am not a computer programmer by training - I'm
an E.E.  The only formal programming training I had in school was with
a language called Fortran.  Heck, most of the stuff I use today didn't
even exist when I was in school.  For the most part, what I've learned
first came from comments by colleagues or references I've run across
reading newsgroups, web pages, technical journals, etc.  I'd then dig
in a bit to find out more and see if it was something that might apply
to what I was doing at work or home.  A small percentage of the time
I'd come across a real "jewel" that then became part of my "toolbox"
going forward.  A perfect example of this is scripting languages, which
I've been using both at work and home for a long time.  Further
disclaimer; everything below is from my perspective as an engineer with
30 years experience designing hardware and occasional software.  Any
omissions or errors are mine and mine alone.


A BIT OF HISTORY:

Scripting is not new; it's been around in various forms almost as long
as the computer itself.  In a nutshell, it's nothing more than
automation - something we all have an interest in or we wouldn't be
tracking this newsgroup.  In this case the automation target is the
computer, with the goal of manipulating it to do things automatically
that otherwise we would have to do manually.  Every type of computer
environment I'm familiar with supports scripting in one form or
another.  In the original DOS environment they were called "batch
files" and, although somewhat primitive compared to scripting languages
in use today, they allowed someone who was not strictly a computer
programmer to be able to put together a "program" that would accomplish
a specific task.  The most common example of this was a file called
"autoexec.bat" which allowed a user to configure his computer the same
way every time it started up.  All you needed was a text editor and you
could edit this file to tailor your computer to the way you wanted.
Many people who did this had limited knowledge of just how computers
worked and didn't even realize that they were writing a computer
program, albeit a simple one.

At about the same time that the PC and DOS were being introduced to the
home computer user, a new operating system called Unix was establishing
itself in the industrial world.  Like other operating systems, it had
an underlying base which controlled the computer, plus a user interface
which allowed the user to interact with it.  Also, like most other
operating systems of the time, the user interface was based on a
command line which allowed someone to type in the operation that was to
be performed.  However, unlike many other operating systems, the Unix
interface was a standalone program called a "shell" with the original
version simply being called "sh" and normally placed in the /bin
directory.  People soon began to tinker with the interface, leading to
a series of improvements that allowed command line editing, a history
buffer of previously executed commands, etc.  And, of course, the
ability to quickly put together a text file of commands to be executed,
called "shell scripts".  With subsequent versions of shells (csh, tcsh,
korn and many others), shell scripts became more sophisticated and
started including concepts that were previously in the domain of
programmming languages, such as control constructs (if/then), looping
(for, while) and the ability to create and use internal variables.
However, unlike most programming languages of the time, these scripts
did not need to be compiled (a process which takes the instructions in
the script text file and turns it into the low-level ones and zeros
that drive the computer hardware).  Instead, these programs relied on
something else to "interpret" them; in this case the interpreter was
the operating system itself.

The concept of interpretation was not new in computer languages; one of
the first languages - Basic - was based on this approach.  Soon the
approch was being applied to a new breed of computer languages.  These
languages consisted of a program which ran under the operating system
of the computer and directly executed the instructions in the user
source text file.  This offered many advantages over traditional
programming:

1) Scripts could be constructed and executed quickly - no lengthly
compile/link operations.

2) A person did not need to be an expert programmer to use them.  In
fact, more often than not, script languages were targeted at
individuals who had a specific task to get done that required the use
of a computer, but they themselves were not necessarily computer
experts.  Nor did they have computer experts at their bidding to write
the program they needed.

3) Scripting languages were usually in the "high level" language
category; they dealt with concepts that were common regardless of the
particular computer or operating system being used.  All you needed was
an interpreter that ran on the specific platform and you could run your
script.  As a bonus, the same script could often run on multiple
platforms with little or no changes.

4) Most of them are open source and can be downloaded for free from a
number of sites.

I first became aware of of this new breed of scripting languages in the
late 1980's.  Like many other engineers, I had a PC at home and was
playing with languages like QuickBasic, which allowed me to create and
compile my own programs that ran under DOS on the PC.  However, the
computer world was undergoing a revolution.  At work I was starting to
use this thing called "X Windows" and, even at home, I found that the
operating system was evolving towards a Windows approach.  I looked at
what it would take to write programs that used this new graphical
approach and quickly became discouraged; hundreds of lines of code just
to pop up a simple window?!  I had become spoiled in being able to
quickly produce a simple program that did what I needed.  Now I was
looking at something that appeared to have a steep learning curve
followed by an increased amount of time to produce the program I
needed.  When I described my problem to a friend at work, he mentioned
that he had just heard about a thing called Tcl/Tk that might be what I
was looking for.  I dug into it and found that Tcl stood for Tool
Control Language and was developed a few years before by John
Ousterhout at the University of California at Berkeley.  Although
initially targeted to ease interfacing with interactive integrated
circuit design tools, Tcl evolved into a general purpose scripting
language.  It was also designed to be extensible, meaning it was
relatively easy to create additional packages that could become part of
the language.  In fact, one of the first extensions was a thing called
Toolkit (Tk - also designed by Ousterhout) which targeted the Graphical
User Interface.  I found that I could write a simple script that would
pop up a window which contained a label with the ubiquitous "Hello,
World!", along with an exit button that would close the window when
clicked.  Oh, happy days!! I'm back in business again!  What was the
effort to write this script?  One line of code!!!  Not only that, but
this script ran both on my workstation at work and my PC at home.
Although differences in the native window managers made the program
look slightly different between the two environments, it basically did
the same thing.  Other scripting languages started appearing around
this time, most notably Perl, and today scripting languages such as
Python and Ruby even incorporate modern concepts such as Object
Orientation.  Although I've played with or worked with most of them, my
favorite still remains Tcl for general purpose work.  It is interesting
to note that many popular extensions that were originally developed for
the Tcl language (such as Tk mentioned above and Expect, an extension
which automates many aspects of software testing) have since been
ported to these newer scripting languages as well.


TCL IN HOME AUTOMATION:

Nearly six years ago, after years of saving my money and dreaming about
home automation, I had a chance to build my "dream house".  The
automation controller was the OnQ 1050 (HAI OmniPro) and initially
allowed me to control security, lighting and HVAC, either through
keypads or a program called PC Access running on my PC.  The PC was
connected to the OnQ controller over an RS-232 link from one of the COM
ports.  Everything was great - until I got my first electric bill.
Over $600!!  This was for a home that was supposed to be energy
efficient; although around 5500 sq. ft. it had concrete form exterior
walls and a host of other characteristics that were supposed to reduce
energy.  The builder was little help ("What did you expect?  It's a big
house and this is the middle of winter"), but I felt something was
wrong.  The HA controller did not log the length of time that a given
HVAC unit was on, something that I felt would help me prove to the
builder that there was a problem with one of the zones.  Somehow, I
needed a way to collect this data and do it fast - the builder was
already most of the way through the punchlist and time was running out
for me.  I had already determined that, although the OnQ didn't
directly tell me whether an HVAC unit was on or off, I could read the
current zone temperature from the thermostat.  I could also read the
current setpoint and, calculating whether the current temperature was
above or below this, I could infer whether the unit was on or off
(actually, there is built-in hysteresis, but this was close enough for
what I needed).  It only took a couple of nights to write and debug the
script and a week to collect enough data to go to the builder and prove
to him that not only was there a problem, but which specific HVAC was
causing it.  Once he replaced the unit the energy usage dropped back to
where I felt it should be.

So far, so good.  Soon I was writing other Tcl scripts for various
purposes.  One collected and logged "event" information such as when
the security alarm was turned on or off, things that would trip an
alarm such as a motion detector tripping or a window opening, etc.
Another allowed more sophisticated control over my newly installed
sprinkler system (ie: don't water if the temperature is below a certain
point or if the moisture sensor indicates the ground is already wet).
These were fun and easy to write but there was one very major drawback;
they all communicated over the single RS-232 link to the OnQ
controller, which could only do one thing at a time.  This meant that
to run any given script the one that was currently running had to be
stopped.  For a while I considered trying to roll everything into one
giant Home Control program but I could see myself constantly tinkering
with it over time as I added new features or upgraded old ones.
Fortunately, Tcl offered the capability of a better solution through
the use of sockets.  For those that are not familiar with them, sockets
offer a method of communication between programs using the same
networking protocols that are used for Internet communication.  From
the user perspective, sockets operate similar to files on a filesystem;
you can open a specific one that you are interested in, read or write
data as long as you want and then close it when you are done.
Communication over sockets can be between two programs running on the
same computer or between separate computers that are connected using
the TCP/IP protocol (usually over wired or wireless Ethernet).  What
this meant was that I could write a single program that would act like
a classic server; it would be the only thing that would communicate
with the OnQ over the RS-232 link.  Various client programs, such as
the sprinkler controller and event logger, would connect to it via the
socket interface and send requests.  These would be queued up by the
server and processed one after the other in the order they were
received.  Associated with each request was the Channel ID (socket)
that the request was received from so that after the request had been
sent to the OnQ for processing and the response received back, it could
then be returned to the correct originating client.  This offered two
big advantages.  First, I could add or modify clients at will without
disturbing a running system.  Second, once I had developed a client, I
could run as many instances as I wanted on any networked platform that
could communicate with the server.  Note that I said platform; as long
as a Tcl/Tk interpreter is available, you should be able to run the
script almost anywhere: MS Windows (from '98 through 2003 Server,
including WinCE), all flavors of Linux, Solaris, HP-UX, etc.  Since
there is an interpreter for my wireless Dell Axim I can even run
clients there (although sometimes I do customize the GUI since there is
a smaller screen area to work with).


HOW TO GET STARTED:

If any of the above sounds interesting and you would like to learn
more, then it's easy to get started.  The first thing you will need is
an interpreter for your platform.  While you can download from many
places, if you are just getting started I would recommend the Active
State distribution (www.activestate.com).  While they do have a number
of tools that they sell, mostly to commercial developers, most of their
scripting distributions are free - you don't even have to fill out any
forms if you don't want to.  Simply click on the language you want -
Tcl is in the upper right-hand corner of the Active State main page -
and you will go to the download page.  Select the package you want (the
free one 8-) ) and click on the Download now link.  This will take you
to the forms page where, if you want to be notified about updates you
can supply contact information.  Click Next and you will go to the page
that will allow you to download the distribution for your platform.
Warning - the Windows package is about 20 Meg so you will want either a
broadband connection or a lot of patience.  Once you have your package
downloaded, the installer makes it a snap and it should be up and
running in less than a minute.

Ready to write your first script?  I told you that you could pop up a
simple window with a single line of code.  Well, here it is.  Open up
your favorite text editor and type in the following:

tk_messageBox -message "Hello, World!" -type ok

Save the file, making sure that it is saved in TEXT mode; for example,
if you are using Wordpad under windows, it will normally try to save
the file in Rich Text Format (RTF) so you will need to select TEXT
format in the "Save as type:" pulldown.  You will also have to make
sure that the file extension is set to ".tcl", for example "hello.tcl".
 Next, open up the file browser for your system and go to the folder
where you just saved the file.  Tcl is commonly pronounced "tickle"
among programmers so the icon is usually shown as a feather.  Double
click on the file and a small window should pop up that says (oddly
enough) Hello, World!  At the bottom of the window is a small button
that says "OK" in the middle.  Click on the button and the window
should close.  Notice, however, that there is still another (blank)
window that popped up on your desktop.  This one should have the name
you gave the script in the title bar.  This is the toplevel window - to
close it you will need to click on the "X" in the upper right-hand
corner.  Next, lets modify our program so that this is done
automatically when you click the OK button.  Start your editor again
and add a second line below the first:

tk_messageBox -message "Hello, World!" -type ok
exit

Previously, when you clicked the OK button, the script advanced from
the first line (which had just finished execution) to the next line.
Well, there was no next line so the interpreter simply waited in the
shell, which was that empty toplevel window that you manually closed.
Run your script again and this time, when you click on the OK button,
the program should completely exit.  Congratulations! You've just
written your first Tcl script.  If you decide to go forward from here,
you will learn that the (blank) toplevel window that popped up along
side your Hello, World window is where your main program will really go
and it is this window that you will resize to whatever size you want
and fill with whatever "widgets" you want to contain in your GUI.

For the next step, I would recommend (if you're using Windows) clicking
on the Start menu and selecting the newly installed Tcl installation.
There should be a section called demos that will show many of the
things that you can do with the language.  Start with the Tk demo and
you will see examples of many of the basic widgets, such as labels,
buttons, listboxes, menus, etc.  The nice thing is that as you select
each one, it will pop up a window showing the example of that widget.
Next, click on the button that says "See Code" and it will pop up the
entire script that it took to create that window.  You will be
astonished to find the amount that you can do with a relatively small
script.

If you're really hooked at this point, your next thought might be to
run out and buy a book on Tcl so that you can really get rolling.
Don't.  You will, if you're still using Tcl in two months, but right
now everything you need is installed right on your computer.  There is
excellant online help that comes with the interpreter along with a host
of examples that will help get you started.  There is also a thriving
user community; the comp.lang.tcl newsgroup is a great place to ask
questions if you get stuck, but often you can answer questions quicker
yourself by checking the Tcler's Wiki at
http://mini.net/cgi-bin/wikit/.  Once you get to a point where a
reference book will help, I would highly recommend "Practical
Programming in Tcl and Tk" by Brent Welch, Ken Jones and Jeff Hobbs.
The current version is the 4th edition but if you can live with a
previous version, I've heard that you can download it for free (Google
can probably point you in the right direction).  Also, although a bit
dated at this point, "Tcl and the Tk Toolkit" by John Ousterhout is the
original documentation and still contains a lot of good information.


CONCLUSIONS:

Rolling your own home automation software may not be for everyone but
if you are interested in customizing your setup, scripting languages
provide capabilities that would otherwise be beyond most of us.  My
most recent home automation addition was whole-house audio
distribution.  I have three of the Channel Plus MDS-6 amplifiers
connected to speakers that were installed when the house was built.
The MDS-6 is controlled via keypads installed in the same general area
as the speakers.  Each keypad communicates over a Cat-5 cable that runs
back to the main amplifier.  All very good, but there is no other
control interface and I wanted a way to tie it into my overall HA
framework.  I designed some hardware that spoofs the MDS-6 into
thinking it's getting commands from a remote keypad, when in reality
the commands are originating on an Atmel AVR microcontroller.  The code
for the AVR is written in "C", but the overall audio interface client
running on the PC (and the Dell Axim) is written in Tcl/Tk.  Without
Tcl to implement the user interface portion, the overall goals  of the
project would have been difficult at best and impossible at worst.
Scripting languages have opened up a whole world of opportunity for me
and, hopefully, they can for you too.