Re: Scripting 101

[nobody@xxxxxxxxxxxx (Dave Houston)]

IMO, any discussion of scripting should begin with a discussion of the
security implications.

     http://research.sun.com/techrep/1997/smli_tr-97-60.pdf

"Dave Harper" <dharper@xxxxxxxxxxxxxxxx> wrote:

>Scripting 101
>
>This post is in response to a query in another thread regarding a
>comment I made about scripting.  I use scripting extensively in my home
>automation environment and it allows me to quickly create software to
>control the diverse hardware I have installed.
>
>First, a disclaimer; I am not a computer programmer by training - I'm
>an E.E.  The only formal programming training I had in school was with
>a language called Fortran.  Heck, most of the stuff I use today didn't
>even exist when I was in school.  For the most part, what I've learned
>first came from comments by colleagues or references I've run across
>reading newsgroups, web pages, technical journals, etc.  I'd then dig
>in a bit to find out more and see if it was something that might apply
>to what I was doing at work or home.  A small percentage of the time
>I'd come across a real "jewel" that then became part of my "toolbox"
>going forward.  A perfect example of this is scripting languages, which
>I've been using both at work and home for a long time.  Further
>disclaimer; everything below is from my perspective as an engineer with
>30 years experience designing hardware and occasional software.  Any
>omissions or errors are mine and mine alone.
>
>
>A BIT OF HISTORY:
>
>Scripting is not new; it's been around in various forms almost as long
>as the computer itself.  In a nutshell, it's nothing more than
>automation - something we all have an interest in or we wouldn't be
>tracking this newsgroup.  In this case the automation target is the
>computer, with the goal of manipulating it to do things automatically
>that otherwise we would have to do manually.  Every type of computer
>environment I'm familiar with supports scripting in one form or
>another.  In the original DOS environment they were called "batch
>files" and, although somewhat primitive compared to scripting languages
>in use today, they allowed someone who was not strictly a computer
>programmer to be able to put together a "program" that would accomplish
>a specific task.  The most common example of this was a file called
>"autoexec.bat" which allowed a user to configure his computer the same
>way every time it started up.  All you needed was a text editor and you
>could edit this file to tailor your computer to the way you wanted.
>Many people who did this had limited knowledge of just how computers
>worked and didn't even realize that they were writing a computer
>program, albeit a simple one.
>
>At about the same time that the PC and DOS were being introduced to the
>home computer user, a new operating system called Unix was establishing
>itself in the industrial world.  Like other operating systems, it had
>an underlying base which controlled the computer, plus a user interface
>which allowed the user to interact with it.  Also, like most other
>operating systems of the time, the user interface was based on a
>command line which allowed someone to type in the operation that was to
>be performed.  However, unlike many other operating systems, the Unix
>interface was a standalone program called a "shell" with the original
>version simply being called "sh" and normally placed in the /bin
>directory.  People soon began to tinker with the interface, leading to
>a series of improvements that allowed command line editing, a history
>buffer of previously executed commands, etc.  And, of course, the
>ability to quickly put together a text file of commands to be executed,
>called "shell scripts".  With subsequent versions of shells (csh, tcsh,
>korn and many others), shell scripts became more sophisticated and
>started including concepts that were previously in the domain of
>programmming languages, such as control constructs (if/then), looping
>(for, while) and the ability to create and use internal variables.
>However, unlike most programming languages of the time, these scripts
>did not need to be compiled (a process which takes the instructions in
>the script text file and turns it into the low-level ones and zeros
>that drive the computer hardware).  Instead, these programs relied on
>something else to "interpret" them; in this case the interpreter was
>the operating system itself.
>
>The concept of interpretation was not new in computer languages; one of
>the first languages - Basic - was based on this approach.  Soon the
>approch was being applied to a new breed of computer languages.  These
>languages consisted of a program which ran under the operating system
>of the computer and directly executed the instructions in the user
>source text file.  This offered many advantages over traditional
>programming:
>
>1) Scripts could be constructed and executed quickly - no lengthly
>compile/link operations.
>
>2) A person did not need to be an expert programmer to use them.  In
>fact, more often than not, script languages were targeted at
>individuals who had a specific task to get done that required the use
>of a computer, but they themselves were not necessarily computer
>experts.  Nor did they have computer experts at their bidding to write
>the program they needed.
>
>3) Scripting languages were usually in the "high level" language
>category; they dealt with concepts that were common regardless of the
>particular computer or operating system being used.  All you needed was
>an interpreter that ran on the specific platform and you could run your
>script.  As a bonus, the same script could often run on multiple
>platforms with little or no changes.
>
>4) Most of them are open source and can be downloaded for free from a
>number of sites.
>
>I first became aware of of this new breed of scripting languages in the
>late 1980's.  Like many other engineers, I had a PC at home and was
>playing with languages like QuickBasic, which allowed me to create and
>compile my own programs that ran under DOS on the PC.  However, the
>computer world was undergoing a revolution.  At work I was starting to
>use this thing called "X Windows" and, even at home, I found that the
>operating system was evolving towards a Windows approach.  I looked at
>what it would take to write programs that used this new graphical
>approach and quickly became discouraged; hundreds of lines of code just
>to pop up a simple window?!  I had become spoiled in being able to
>quickly produce a simple program that did what I needed.  Now I was
>looking at something that appeared to have a steep learning curve
>followed by an increased amount of time to produce the program I
>needed.  When I described my problem to a friend at work, he mentioned
>that he had just heard about a thing called Tcl/Tk that might be what I
>was looking for.  I dug into it and found that Tcl stood for Tool
>Control Language and was developed a few years before by John
>Ousterhout at the University of California at Berkeley.  Although
>initially targeted to ease interfacing with interactive integrated
>circuit design tools, Tcl evolved into a general purpose scripting
>language.  It was also designed to be extensible, meaning it was
>relatively easy to create additional packages that could become part of
>the language.  In fact, one of the first extensions was a thing called
>Toolkit (Tk - also designed by Ousterhout) which targeted the Graphical
>User Interface.  I found that I could write a simple script that would
>pop up a window which contained a label with the ubiquitous "Hello,
>World!", along with an exit button that would close the window when
>clicked.  Oh, happy days!! I'm back in business again!  What was the
>effort to write this script?  One line of code!!!  Not only that, but
>this script ran both on my workstation at work and my PC at home.
>Although differences in the native window managers made the program
>look slightly different between the two environments, it basically did
>the same thing.  Other scripting languages started appearing around
>this time, most notably Perl, and today scripting languages such as
>Python and Ruby even incorporate modern concepts such as Object
>Orientation.  Although I've played with or worked with most of them, my
>favorite still remains Tcl for general purpose work.  It is interesting
>to note that many popular extensions that were originally developed for
>the Tcl language (such as Tk mentioned above and Expect, an extension
>which automates many aspects of software testing) have since been
>ported to these newer scripting languages as well.
>
>
>TCL IN HOME AUTOMATION:
>
>Nearly six years ago, after years of saving my money and dreaming about
>home automation, I had a chance to build my "dream house".  The
>automation controller was the OnQ 1050 (HAI OmniPro) and initially
>allowed me to control security, lighting and HVAC, either through
>keypads or a program called PC Access running on my PC.  The PC was
>connected to the OnQ controller over an RS-232 link from one of the COM
>ports.  Everything was great - until I got my first electric bill.
>Over $600!!  This was for a home that was supposed to be energy
>efficient; although around 5500 sq. ft. it had concrete form exterior
>walls and a host of other characteristics that were supposed to reduce
>energy.  The builder was little help ("What did you expect?  It's a big
>house and this is the middle of winter"), but I felt something was
>wrong.  The HA controller did not log the length of time that a given
>HVAC unit was on, something that I felt would help me prove to the
>builder that there was a problem with one of the zones.  Somehow, I
>needed a way to collect this data and do it fast - the builder was
>already most of the way through the punchlist and time was running out
>for me.  I had already determined that, although the OnQ didn't
>directly tell me whether an HVAC unit was on or off, I could read the
>current zone temperature from the thermostat.  I could also read the
>current setpoint and, calculating whether the current temperature was
>above or below this, I could infer whether the unit was on or off
>(actually, there is built-in hysteresis, but this was close enough for
>what I needed).  It only took a couple of nights to write and debug the
>script and a week to collect enough data to go to the builder and prove
>to him that not only was there a problem, but which specific HVAC was
>causing it.  Once he replaced the unit the energy usage dropped back to
>where I felt it should be.
>
>So far, so good.  Soon I was writing other Tcl scripts for various
>purposes.  One collected and logged "event" information such as when
>the security alarm was turned on or off, things that would trip an
>alarm such as a motion detector tripping or a window opening, etc.
>Another allowed more sophisticated control over my newly installed
>sprinkler system (ie: don't water if the temperature is below a certain
>point or if the moisture sensor indicates the ground is already wet).
>These were fun and easy to write but there was one very major drawback;
>they all communicated over the single RS-232 link to the OnQ
>controller, which could only do one thing at a time.  This meant that
>to run any given script the one that was currently running had to be
>stopped.  For a while I considered trying to roll everything into one
>giant Home Control program but I could see myself constantly tinkering
>with it over time as I added new features or upgraded old ones.
>Fortunately, Tcl offered the capability of a better solution through
>the use of sockets.  For those that are not familiar with them, sockets
>offer a method of communication between programs using the same
>networking protocols that are used for Internet communication.  From
>the user perspective, sockets operate similar to files on a filesystem;
>you can open a specific one that you are interested in, read or write
>data as long as you want and then close it when you are done.
>Communication over sockets can be between two programs running on the
>same computer or between separate computers that are connected using
>the TCP/IP protocol (usually over wired or wireless Ethernet).  What
>this meant was that I could write a single program that would act like
>a classic server; it would be the only thing that would communicate
>with the OnQ over the RS-232 link.  Various client programs, such as
>the sprinkler controller and event logger, would connect to it via the
>socket interface and send requests.  These would be queued up by the
>server and processed one after the other in the order they were
>received.  Associated with each request was the Channel ID (socket)
>that the request was received from so that after the request had been
>sent to the OnQ for processing and the response received back, it could
>then be returned to the correct originating client.  This offered two
>big advantages.  First, I could add or modify clients at will without
>disturbing a running system.  Second, once I had developed a client, I
>could run as many instances as I wanted on any networked platform that
>could communicate with the server.  Note that I said platform; as long
>as a Tcl/Tk interpreter is available, you should be able to run the
>script almost anywhere: MS Windows (from '98 through 2003 Server,
>including WinCE), all flavors of Linux, Solaris, HP-UX, etc.  Since
>there is an interpreter for my wireless Dell Axim I can even run
>clients there (although sometimes I do customize the GUI since there is
>a smaller screen area to work with).
>
>
>HOW TO GET STARTED:
>
>If any of the above sounds interesting and you would like to learn
>more, then it's easy to get started.  The first thing you will need is
>an interpreter for your platform.  While you can download from many
>places, if you are just getting started I would recommend the Active
>State distribution (www.activestate.com).  While they do have a number
>of tools that they sell, mostly to commercial developers, most of their
>scripting distributions are free - you don't even have to fill out any
>forms if you don't want to.  Simply click on the language you want -
>Tcl is in the upper right-hand corner of the Active State main page -
>and you will go to the download page.  Select the package you want (the
>free one 8-) ) and click on the Download now link.  This will take you
>to the forms page where, if you want to be notified about updates you
>can supply contact information.  Click Next and you will go to the page
>that will allow you to download the distribution for your platform.
>Warning - the Windows package is about 20 Meg so you will want either a
>broadband connection or a lot of patience.  Once you have your package
>downloaded, the installer makes it a snap and it should be up and
>running in less than a minute.
>
>Ready to write your first script?  I told you that you could pop up a
>simple window with a single line of code.  Well, here it is.  Open up
>your favorite text editor and type in the following:
>
>tk_messageBox -message "Hello, World!" -type ok
>
>Save the file, making sure that it is saved in TEXT mode; for example,
>if you are using Wordpad under windows, it will normally try to save
>the file in Rich Text Format (RTF) so you will need to select TEXT
>format in the "Save as type:" pulldown.  You will also have to make
>sure that the file extension is set to ".tcl", for example "hello.tcl".
> Next, open up the file browser for your system and go to the folder
>where you just saved the file.  Tcl is commonly pronounced "tickle"
>among programmers so the icon is usually shown as a feather.  Double
>click on the file and a small window should pop up that says (oddly
>enough) Hello, World!  At the bottom of the window is a small button
>that says "OK" in the middle.  Click on the button and the window
>should close.  Notice, however, that there is still another (blank)
>window that popped up on your desktop.  This one should have the name
>you gave the script in the title bar.  This is the toplevel window - to
>close it you will need to click on the "X" in the upper right-hand
>corner.  Next, lets modify our program so that this is done
>automatically when you click the OK button.  Start your editor again
>and add a second line below the first:
>
>tk_messageBox -message "Hello, World!" -type ok
>exit
>
>Previously, when you clicked the OK button, the script advanced from
>the first line (which had just finished execution) to the next line.
>Well, there was no next line so the interpreter simply waited in the
>shell, which was that empty toplevel window that you manually closed.
>Run your script again and this time, when you click on the OK button,
>the program should completely exit.  Congratulations! You've just
>written your first Tcl script.  If you decide to go forward from here,
>you will learn that the (blank) toplevel window that popped up along
>side your Hello, World window is where your main program will really go
>and it is this window that you will resize to whatever size you want
>and fill with whatever "widgets" you want to contain in your GUI.
>
>For the next step, I would recommend (if you're using Windows) clicking
>on the Start menu and selecting the newly installed Tcl installation.
>There should be a section called demos that will show many of the
>things that you can do with the language.  Start with the Tk demo and
>you will see examples of many of the basic widgets, such as labels,
>buttons, listboxes, menus, etc.  The nice thing is that as you select
>each one, it will pop up a window showing the example of that widget.
>Next, click on the button that says "See Code" and it will pop up the
>entire script that it took to create that window.  You will be
>astonished to find the amount that you can do with a relatively small
>script.
>
>If you're really hooked at this point, your next thought might be to
>run out and buy a book on Tcl so that you can really get rolling.
>Don't.  You will, if you're still using Tcl in two months, but right
>now everything you need is installed right on your computer.  There is
>excellant online help that comes with the interpreter along with a host
>of examples that will help get you started.  There is also a thriving
>user community; the comp.lang.tcl newsgroup is a great place to ask
>questions if you get stuck, but often you can answer questions quicker
>yourself by checking the Tcler's Wiki at
>http://mini.net/cgi-bin/wikit/.  Once you get to a point where a
>reference book will help, I would highly recommend "Practical
>Programming in Tcl and Tk" by Brent Welch, Ken Jones and Jeff Hobbs.
>The current version is the 4th edition but if you can live with a
>previous version, I've heard that you can download it for free (Google
>can probably point you in the right direction).  Also, although a bit
>dated at this point, "Tcl and the Tk Toolkit" by John Ousterhout is the
>original documentation and still contains a lot of good information.
>
>
>CONCLUSIONS:
>
>Rolling your own home automation software may not be for everyone but
>if you are interested in customizing your setup, scripting languages
>provide capabilities that would otherwise be beyond most of us.  My
>most recent home automation addition was whole-house audio
>distribution.  I have three of the Channel Plus MDS-6 amplifiers
>connected to speakers that were installed when the house was built.
>The MDS-6 is controlled via keypads installed in the same general area
>as the speakers.  Each keypad communicates over a Cat-5 cable that runs
>back to the main amplifier.  All very good, but there is no other
>control interface and I wanted a way to tie it into my overall HA
>framework.  I designed some hardware that spoofs the MDS-6 into
>thinking it's getting commands from a remote keypad, when in reality
>the commands are originating on an Atmel AVR microcontroller.  The code
>for the AVR is written in "C", but the overall audio interface client
>running on the PC (and the Dell Axim) is written in Tcl/Tk.  Without
>Tcl to implement the user interface portion, the overall goals  of the
>project would have been difficult at best and impossible at worst.
>Scripting languages have opened up a whole world of opportunity for me
>and, hopefully, they can for you too.