...
Table of Contents maxLevel 3 minLevel 2
...
| Anchor | ||||
|---|---|---|---|---|
|
The Real Time Picture Processor (RTPP) was one of the first special-purpose hardware computers developed for grayscale image processing and was designed to aid in biological image analysis. It was developed at the National Cancer Institute (NCI) of the National Institutes of Health (NIH).
...
There are two major goals of this history: to document the events and conditions that led to the creation of one of the first grayscale image processors, and to describe the highly effective complementary collaboration that allowed this project to flourish. Occasionally, references will be made to other later advances indirectly related to the RTPP work that would not have happened without the RTPP. Where possible, we have linked to open access journal PDFs, and have included PDFs of the key technical reports describing the RTPP on this Web site.
| Anchor | ||||
|---|---|---|---|---|
|
The birth of the concept of the RTPP
...
The first thing he wanted to be able to do was move a slide via a computer-controlled microscope stage. Initially, he was going to do it with analog feedback. He talked to Wes Clark (who had helped build the LINC computer with Charlie Molner and others). Wes convinced Lew that he really wanted a digital stage - not an analog one - so that is what Lew developed: a series of stepping-motor-controlled stages that improved with each generation. The original design connected the stage with rubber bands, which was then greatly improved with direct stepping-motor drives. Lew had also been working with Russell Kirsch and Bill Watt from the National Bureau of Standards (NBS, now the National Institute of Standards and Technology or NIST 
). This early work involved describing biological images using computer picture grammars [1] that attempted to bring artificial intelligence and algorithmic methods to the description of biological images.
| Anchor | ||||
|---|---|---|---|---|
|
Evolution of the computer-controlled microscope
...
We left NINDB for the IPU in the early 1970s and became the Image Processing Section (IPS) at NCI about 1980, and moved to the Park Building, Parklawn Drive, in Rockville. Although the members of the IPS went on to work in other areas not directly related to the RTPP, this history will concentrate on the work that was directly related to the RTPP.
| Anchor | ||||
|---|---|---|---|---|
|
The unique RTPP parallel-bus architecture (at that time)
...
Scientists used the RTPP as finally constructed to analyze data in a variety of biomedical domains including optical microscope images of optical serial sections of brain tissue, stained bone marrow smears, and tissue cultures using phase contrast and differential interference optics. The latter was used in tracking cell membrane extents of macrophages in tissue culture over time as the cells tried to phagocytize various types of asbestos fibers. The goal was to better understand fiber carcinogenicity and the dynamics of fiber ingestion [8, 9, 10]. The bone marrow smear image analysis was part of my Ph.D. dissertation [11, 12, TR-653, TR-655]. The RTPP was also used for 2D electrophoretic gel images for a variety of biological materials [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32], and for RNA electron micrographs of secondary structure, which was part of Bruce Shapiro's Ph.D. dissertation [33, 34, 35, 36, 37, 38, TR-BAS78].
Figure 1. Dr. Lewis Lipkin headed the project. His group started working on computer-controlled optical microscopy in the Perinatal Research Branch (PRB) of NINDB. The group later changed its name and institutes to the Image Processing Unit (IPU) in NCI in the Laboratory of Pathology. IPU later became the Image Processing Section (IPS) in NCI. The Section later became part of the Laboratory of Mathematical Biology (LMMB) in NCI under Dr. Charles DeLisi, Ph.D., and still later under Dr. Jacob Maizel, Ph.D.. The laboratory changed its name to the Laboratory of Experimental and Computational Biology (LECB) under Jake Maizel. The laboratory is currently refocused on nanobiology and is now called the Center for Computer Research Nanobiology Program (CCRNP) directed by Dr. Robert Blumenthal, Ph.D.(CCRNP has an additional research Web site).
| Anchor | ||||
|---|---|---|---|---|
|
2. The Real Time Picture Processor Development Team
The following individuals were involved with the design and development of the RTPP. The history goes into the who, what, where, when, and why. The second list enumerates some of the individuals doing biomedical research in which the RTPP played a role.
| Anchor | ||||
|---|---|---|---|---|
|
- Lewis Lipkin, M.D., (mathematics and physical chemistry, and a neuropathologist), Head of the Image Processing Section (IPS); previously the (PRB, NINDB) and then the Image Processing Unit (IPU) in the NCI.
- Peter Lemkin, Ph.D. & M.S. EE, computer scientist and electrical engineer, IPS/NCI, and previously in (PRB, NINDB) and in IPU/NCI
- George Carman, M.S. EE, electrical engineer and computer hardware architecture, Technical Development Section (TDS), NINDB; Carman Engineering (now Lucidyne Corp ).
- Morton Schultz, B.S. EE, electrical engineer, IPS/NCI, and previously in IPU/NCI
- Bruce Shapiro, Ph.D., B.S. math & physics, computer scientist, IPS/NCI, and previously in (PRB, NINDB),and in IPU/NCI
- Sprague Hazard, mechanical engineer (contractor consultant)
- Peter Kaiser, B.S. CS, computer scientist (IPU) in the NCI
- Earl Smith, M.S. CS, computer scientist (IPU) in the NCI
- Dan Kilgore, B.S. EE, computer programmer [Digital Equipment Corp (DEC) software engineer]
- Tom Duval and later Jim Camper, electronics technicians - helped construct the RTPP racks, and power-supplies cabinets
- Cambion Corporation, wire-wrapped the remaining 63 buffer memory boards and the back-planes
| Anchor | ||||
|---|---|---|---|---|
|
Where: PRB was the Perinatal Research Branch in the National Institute of the Neurological Disease and Blindness (NINDB). IPU was the Image Processing Unit of the National Cancer Institute (NCI). The IPU later became the Image Processing Section (IPS).
| Anchor | ||||
|---|---|---|---|---|
|
Who: The initial participants were Dr. Lewis Lipkin, myself, and George Carman. Later in the process, Morton Schultz and Bruce Shapiro joined the design group. Peter Kaiser and Earl Smith participated for a few years. During this time, Bruce Shapiro and I were part-time Ph.D. students in the Computer Science Department of the University of Maryland with Professor Azriel Rosenfeld, one of the early leaders in the field of image processing. Both Bruce [35, 37, 38, 39, TR-BAS78] and I [8,12, TR-653, TR-655] wrote dissertations on image processing. We were able to combine some of our applied NIH research as part of our Ph.D. research and use what we were learning about image processing and computer science to our NIH research. George was able to apply many of the ideas he had learned with his masters in Computer Hardware Architecture.
| Anchor | ||||
|---|---|---|---|---|
|
Major RTPP users and their biomedical research
...
The Real Time Picture Processor project was initiated after the successful completion of another project, the National Cancer Institute (NCI) autoradiograph grain counter [2]. This was one of the first computer-controlled microscopes (from the "NIH Record," about 1974). At the time, fluorescent antibodies were not commonly used for quantifying metabolism, so cell metabolism was often measured using autoradiography methods. When cells were grown in tissue culture with 3H-radiolabeled media, the radioactivity incorporated into cells could be used to estimate their metabolism. Dried slides of the cell culture sample were coated with photographic emulsion and exposed for weeks to months in the dark. They were then photographically developed making the silver grains embedded in the emulsion visible. Grains could be tracked and uniquely counted by serially focusing through the emulsion as individual grains were followed. The number of grains was proportional to the amount of 3H-radiolabeled media taken up by the cells and was a quantitative measurement of metabolism. The user would select a set of cells to be counted, adding them to the pick-list, and then let the machine automatically revisit the cells and make the grain counting measurements.
| Anchor | ||||
|---|---|---|---|---|
|
The grain counter system was constructed by George Carman, an electrical engineer who was finishing up at the National Institute of Neurological Disease and Blindness (NINDB) at the time and later worked on the project under contract; me, an electrical engineer who was starting a part-time Ph.D. program in computer science; and Lewis "Lew" Lipkin, a neuropathologist with extensive computer software and hardware experience. Lew's group was transitioning from NINDB to NCI. George was in the Technical Development Section (TDS) at NINDB under Jim Bryan and Ted Coburn, who were experts in analog circuits. The TDS provided all types of electronics and mechanical engineering support for NINDB from basic design services to machine shops that helped us enormously. Sprague Hazard was one of their expert mechanical engineering consultants and later became one of ours. George had just finished his masters in computer hardware architecture before coming to NINDB and was well versed in digital circuits. Lew contacted the TDS to help with the digital logic for the early stepping-motor-controlled digital stage and George responded. Lew and George hit it off very well, as each understood and appreciated the other's area of expertise. George had just finished a project for TDS working with Jim Bryan that involved capturing XY coordinates from an analog TV camera so they could be recorded on a DEC PDP12 computer. When Lew described the grain counting problem and discussed the capturing of XY coordinates with George for the future grain counter, George already knew how to do it. That's how it started.
The grain counter consisted of a small Quantimet image processing system, a Zeiss Isoplan microscope with a Zeiss XY-axes stepping-motor controlled stage and a Z-axis (i.e., focus) stepping-motor-control that we added. The mechanical setup was designed with the help of the TDS machine shop. A PDP8e computer was interfaced to this data acquisition hardware and stepping-motor (X,Y,Z)-axes control. The operator would find a radio-labeled cell using a joystick (X,Y) control and a Z-axis controlled stage. They would then initiate a data capture of the (X,Y) coordinates of all of the silver grains in the field into a hardware shift register. This would then be transferred to the PDP8e for each 0.2 micron steps in the Z-axis as the microscope focused up and down for thick sections. The grains were tracked between optical serial sections and the actual count of grains for the entire cell recorded without double-counting grains.
| Anchor | ||||
|---|---|---|---|---|
|
We were able to easily control the microscope and process the data using a small amount of PDP8e memory because of the computer language used. DEC's Fortran-II software language compiler running under their OS/8 operating system for the PDP8e allowed the insertion of assembly language that could reference I/O instructions (called IOPs) directly and could also directly reference Fortran variables. This made programming our new hardware relatively easy to do.
| Anchor | ||||
|---|---|---|---|---|
|
The following is an example of Fortran-II mixed code from the RTPP BMOMNI I/O software library (used to access the RTPP hardware from the PDP8e). For those interested, several technical reports available in this history describe the RTPP I/O instructions and design in more detail [TR-7, TR-7a, TR-21, TR-22, TR-23]. The "S" in column 1 indicates that that line should be treated as assembly language; an assembly code variable with a "\" in front of it indicates it is a Fortran variable. The same code style was used with the grain counter as with the RTPP. On the surface, Fortran-II was not a very powerful language, but the combination of these two features made it ideal for easily programming special purpose hardware. We had learned how to control hardware from the software for the grain counter, so that hurdle was already solved when we tackled the RTPP hardware/software-interfacing problem. The success of this hardware/software/microscope system gave us the confidence to go to the next level, a general-purpose image-processing computer that was the RTPP. There is more discussion and the BMON2 source code later in this history.
The plan was to have the NCI replicate these grain counter systems in three or four grantee laboratories. We had put out bids for the replication of the system. But, as with many technological break-throughs, the system worked, but better, less-expensive methods using new antibody and flow cytometry methods were becoming available. So autoradiography was replaced by other systems for measuring and quantifying specific cell types where tracking individual silver grains was not required. The additional grain counters were never built.
4.
| Anchor | ||||
|---|---|---|---|---|
|
The Real Time Picture Processor consisted of the following integrated components:
...
The block diagrams for this configuration are shown in Figures 4 and 5. The Quantimet was designed to perform simple binary thresholding of video analog data and counting operations on the non-interlaced analog video signal, but could not perform complex grayscale operations such as neighborhood computations. Later the RTPP/PDP8e system was interfaced to a DECsystem-2020 computer running the TOPS-10 operating system. Image acquisition and user interaction were relegated to the RTPP/PDP8e while complex analyses were done on the DECsystem-2020. Many of the figures illustrating the RTPP were drawn by Jo Abbott, our secretary and graphics draftsperson, during the initial design phase before we moved to the Park Building.
| Anchor | ||||
|---|---|---|---|---|
|
The Quantimet plumbicon analog video camera was attached to a computer-controlled Zeiss Axiomat microscope with (X,Y)-axes (0.5 micron/steps) Zeiss stepping-motor stage, and Z-axis focus (0.2 micron/step) stepping-motor controlled by the PDP8e. The anti-backlash Z-axis stepping-motor control-assembly was added with the help of Sprague Hazard (the same TDS consultant we had used with the grain counter project), and constructed by the NINDB machine shop. The advantage of Hazard's brilliant anti-backlash Z-axis design was that by moving past the point of interest and then reapproaching it from the same direction each time, one minimized mechanical hysteresis so that random accessed points of the slide could be repositioned quite reliably in three dimensions. Lew's idea of the slide as a 2D array information resource had been expanded to a 3D array (X,Y,Z). The Quantimet analog vidicon camera was used with regular 35mm camera lenses with a uniform illumination light-box for scanning 2D electrophoretograms, electron micrographs of RNA molecules, and other image sources (see figure 18). It was used with a variety of normal, wide-angle and macro-zoom lenses depending on the material we were investigating. The cameras could be easily switched. The plumbicon had a more linear and wider dynamic range and was better suited for microscopy.
| Anchor | ||||
|---|---|---|---|---|
|
The RTPP digital image capture system was called the image "buffer memory" and could capture images directly from the selected Quantimet video camera. Using a $10,000 high-speed analog-to-digital converter (A/D) that was the size of a large DVD player, the analog video was digitized to be captured into the image buffer memories. Similarly, the digital output from the buffer memory was displayed on the Quantimet TV monitor through a high-speed digital-to-analog (D/A) converter. The RTPP system could display the live video or video derived from the image buffer memory on a high-resolution (860x720 pixels - high resolution at that time) non-interlaced TV display. This part of the system was completed and used in many projects. Today, digital cameras and cell phones with cameras have similar A/D and D/A capabilities for under $100.
Because the buffer memories were easily random accessed (one of the advantages of using RAMs instead of shift-register chips), it was in effect a 1-megabyte extension of the PDP8e memory that had a maximum of 32K 12-bit words. We used this to advantage when constructing various complex segmentation and spot pairing algorithms implementing linked-lists stored in the buffer memories. Occasionally, we would drop a bit in the buffer memory and the program would crash or go into an infinite loop as the linked-list got corrupted. However, on the whole, the buffer memory design was state-of-the-art at the time and allowed many of the applications we had created to work as well and to push the envelope on what could be done.
| Anchor | ||||
|---|---|---|---|---|
|
The second phase of the RTPP was the design of a special-purpose 48-bit triple-operand, real-time computer processor called the General Picture Processor or GPP. This GPP would perform parallel image processing operations on 3x3 pixel neighborhoods in the buffer memory throughout the selected images. The GPP design had two input operands and one output operand. Each operand was assigned to an image buffer (there were sixteen 256x256 8-bit pixels per image buffer). The GPP included 3x3 pixel triple operand instructions, which would tessellate over the entire 256x256 pixel image space. The design is described in [3-4, TR-7, TR-7a, TR-22]. A software assembler for the GPP instruction set (GPPASM) [TR-16] and a debugger (DDTG) [TR-2] for the GPP were written on the PDP8e and ready to use with the hardware when it was built. The GPP hardware part of the RTPP was not completed due to a shift in NCI budget priorities. Considering the exponential increase over time in computing power of general-purpose microprocessors as well as their greatly decreasing cost, this was probably a wise decision. It became clear that software efforts would be more effective for many (but certainly not all!) problems. The paradigm had shifted.
The design of the RTPP was presented at the 1973 Asilomar Third Engineering Foundation Conference on Automated Cytology and published in 1974 [3-4]. This conference and a subsequent automated cytology workshop concentrated on the two solutions then available: image processing and pattern recognition of cell images, and the evolving field of flow cytometry. NIH was funding both fields. During this time we developed plans for integrating artificial intelligence techniques for understanding and analyzing biological materials and systems incorporating the RTPP, and these were also presented at the Asilomar workshop [5, TR-15].
5.
| Anchor | ||||
|---|---|---|---|---|
|
The project was started about 1972. By this time, George Carman had left the National Institute of Neurological Disease and Blindness (NINDB), moved to Oregon and was working under contract with our National Cancer Institute (NCI) group. I had been working on a Ph.D. in computer science at the University of Maryland with specialization in image processing and so had Bruce Shapiro. So the General Picture Processor (GPP) design reflected many of the requirements of image processing methods. Lewis "Lew" Lipkin, with his broad understanding of image processing, was also heavily involved in the design. Lew, Bruce, and I would discuss the types of image processing operations we required in brain storming sessions. Then, George and I would have long phone-conversation design sessions where I described the image processing needs discussed in the local Image Processing Unit's group design sessions to George who then worked out the details on how to implement the required operations in the hardware design. I documented these design sessions, which resulted in the technical reports [TR-2, TR-7, TR-7a, TR-16, TR-21, TR-21b,TR-22].
The hardware system design was a joint effort with primary hardware electronics design by George Carman and primary software design by me. The RTPP electronics design was incrementally created in many hours-long phone conferences between George and myself discussing and negotiating requirements for image processing, possible implementations, implications of the designs for hardware and for software, etc. These long, detailed discussions reviewed and modified our snail-mailed blueprints and design documents (this was before e-mail and common access to the Internet). Our phone sessions allowed the iteration, refinement, and extension of the design to take into account the difficulty of programming the proposed hardware and the difficulty and expense of building the hardware. This joint design also allowed the IPU (me in particular) to start building the PDP8e software to interface with the hardware before the RTPP was delivered. In the end, both goals were optimized and the system worked. Some of this process was described in [3-4] and a few of the critical design ideas are listed in this history in some of the figures.
| Anchor | ||||
|---|---|---|---|---|
|
Once George wired and debugged one of the buffer image memory boards, we had a contractor, Cambion Corporation, build the remaining 63 boards (see an example of a board in figures 8 through 10 below). Each board consisted of 64 4K-bit dynamic RAMs (Random Access Memory chips). Four boards implemented a 256x256 pixel by two 8-bit bytes sub-image. These were among the first "high" density memory chips available at the time. Of course being the first generation of a high-density dynamic RAM chip, they had a high failure rate. So George built memory-testing software on the PDP8e that could pinpoint a bad chip on a particular board enabling us to unplug the bad chip and replace it with a new one. This saved a huge amount of time in finding the bad chips and helped improve uptime of the RTPP during its lifetime.
Not only did Cambion build the buffer memory boards, but using their standard technology, they also wired much of the backplanes. Their technology was an integrated system, and had been developed for rapid prototype construction in aerospace projects. It included cards, sockets, and racks. The system would not have worked had the parts been obtained from different vendors. By adapting Cambion's standards, we were able to take advantage of the reliability Cambion had developed for this kind of work. After Cambion created the boards and backplanes, our electronics technicians assembled them into several cabinets of 19" vertical racks including one cabinet for the power supplies. The buffer memories were 16 cards to a rack, with four racks. To avoid overheating, the cards were inserted in every other slot. Then the equipment was shipped to George in Oregon to finish construction and debugging. We had purchased a PDP8e for him to use in developing, debugging, and testing the interface. The computer was also critical for George to create various software tools to help manage the project. These included a wirewrap database program that could take pairs of (drawing #, chip #, pin #) triples that indicated a pair of wires to be connected using a technique called "wirewrap." This methodology was critical since a single buffer memory card was described in a large number of blueprints and it would be difficult to keep straight which pins connected to other pins in this complex global diagram. George then wrote additional software to translate these pairs to the standard lists that Cambion required. In a biomedical image processing and electronics conference, George's triple notation and his new way to handle the increasing complexity of multiple drawing wiring lists received a good reception from some of the developers of VHDL (a hardware description language). Because of space limitations, George put the PDP8e into a closet of his house with additional AC cooling. The PDP8e at that time cost more than his house. Today, the most inexpensive computers are many orders of magnitude more powerful than the PDP8e at a small fraction of their cost.
| Anchor | ||||
|---|---|---|---|---|
|
After George got major parts of the system working in Oregon, he shipped the multiple electronics racks to Building 36 in Bethesda about 1976. He flew in for a week to help debug the initial buffer memories, the RTPP controller, and the connection to the Quantimet both for display and data capture. George and Mort Schultz, did a marathon week of late-night engineering debugging to work out some of the final kinks. Many pizzas helped fuel these sessions. I had been working in parallel on the software interface called BMON (Buffer Memory Monitor System), which was then tested against the hardware to see if it worked more or less according to the design. Hardware and software were iteratively changed as required to accommodate each other.
...
A more robust version of the control software was called BMON2 (Buffer Memory Monitor System) was written and used to interact with the RTPP. It integrated other programs and scripts that analyzed data from the RTPP [40, TR-21, TR-21b, TR-23]. BMON2 was written in Fortran-II under the PDP8e OS/8 operating system. As with the grain counter project, the ability to mix assembly language in with the Fortran allowed easy control of the more than 100 hardware instructions that we added to the RTPP controller (See [TR-7a] for details).
| Anchor | ||||
|---|---|---|---|---|
|
Under Lew's direction - and based on his long experience as a microscopist - the microscope concepts evolved over several generations of computer-controlled designs. The engineering machine shop in NINDB in Building 36 constructed the microscope assembly for the NCI grain counter project. They had an outstanding mechanical engineer consultant, Sprague Hazard, who previously solved some of the very tricky issues including removing the hysteresis in the Z-axis stepping-motor control for the grain counter microscope. He designed additional hardware for the microscope using anti-backlash gears with an approach similar of running the stepping motorsthat we had used in the grain counter. We used this method in the commercial X,Y microscope stepping-motor stage. He also designed a color-filter changer that implemented Lew's insistence on the importance of monochromatic light in micrographic analysis. The changer would swap interference filters in the light path. These successful experiences in constructing the grain counter were then leveraged when we built a new microscope around the Zeiss Axiomat for the RTPP - again with the help of Sprague Hazard who incorporated some very creative ideas.
...
In retrospect, we made one mistake in designing the optical microscope. It should have been an inverted microscope from the start because most of our efforts dealt with living cultures. It was difficult to do a living culture. To put a tissue culture plate down on the stage, you had to use inverted objectives because of the standard microscope structure. The optical path was such that the index of refraction of the culture flask introduced so much of an optical path that only the lower power inverted objectives could be used.
| Anchor | ||||
|---|---|---|---|---|
|
We had started constructing the RTPP when we were in Building 36 (NINDB's building) in Bethesda and around 1980, we were moved to the off-campus Park Building because we had switched from NINDB to NCI, and NCI had no extra lab space available on the Bethesda campus. In moving, we had to dissemble the RTPP and PDP8e racks, Quantimet, Axiomat microscope, etc., and reassemble the system at the Park building. With George's help (he flew in again), Mort got the system back on the air within a reasonable time.
...
The Park Building was less than ideal, because of frequent loss of power, loss of air-conditioning (the temperature rising to over 100 degrees F one time destroyed several boards in the DECsystem-2020), and roof-leaks on the computers whenever it rained hard. In short, it was a terrible building. We had mix-ups including advertising mail addressed to "Mr. Park Building." But, during the time in the Park Building, we began to use the RTPP more heavily and then built a second stripped-down version of the RTPP with the same type of PDP8e and image buffer memories, but with a Conrac TV monitor, graph-pen tablet (for entering x,y, data), and without the fancy control desk.
...
| Anchor | ||||
|---|---|---|---|---|
|
The software control program for the buffer memory I constructed on the PDP8e was called BMON2 (the Buffer Memory Monitor System) [40, TR-21, TR-21b, TR-23] and written in Fortran-II. BMON2, in addition to interfacing with the RTPP, also allowed running other programs to be batched to analyze the data. Given that the PDP8e had 32K words of memory, this was critical for doing complex sequential operations and for easily writing new RTPP applications. A Fortran-II library that could interface with the RTPP, BMOMNI [TR-23], allowed these other programs to access the RTPP as required. (See discussion on Fortran-II in the section on the grain counter. This shows the BMOMNI Fortran code.) BMON2 could capture and display images and do many image processing operations on the PDP8e. Another program called FLICKER [13] ran on the PDP8e and was used to analyze 2D gel images visually by alternately displaying one movable image on the video screen relative to another that was held in a constant screen position. Later, it allowed the comparison of two saved images as well. So a set of images could be compared against a reference sample. Some of the ideas on using flickering images to detect subtle differences in image matching were suggested by Bernice Lipkin, who is an expert in psychopictorics [41]. A third-generation version of FLICKER is available as open-source software at http://open2dprot.sourceforge.net/Flicker .
| Anchor | ||||
|---|---|---|---|---|
|
For those interested in how we coded various image processing functions, we have an annotated list of the BMON2 Fortran-II programs and libraries. If you look at this, you might want also to take a look at the associated paper and technical reports on BMON2 which describe the design in more detail [40, TR-21, TR-21b, TR-23].
| Anchor | ||||
|---|---|---|---|---|
|
A Digital Equipment Corporation DECsystem-2020 was installed in the laboratory after we moved to the Park Building. We had been using the NIH's Division of Computer Research and Technology (DCRT) [now the Center for Information Technology (CIT)] DECsystem-10 time-shared system. As we used TOPS-10 operating system on the DECsystem-10, we installed TOPS-10 on the new DECsystem-2020. Bruce Shapiro had implemented a message-switching high-speed 9600-baud (normal speed was 300 or 1200 baud at the time) serial line multiplexor so we could move images and data to/from the DCRT system. However, the costs for the increasing amount of time we used on the DCRT system was escalating. For a cost comparable to renting time over a few years, we could purchase a dedicated system and have more compute power as well. So NCI supported us in purchasing the DECsystem-2020. This was a DEC Unibus system, which meant we could interface our hardware to this then-powerful 36-bit computer. In hindsight, this was one of the best procurements that Lew made. It offered us vastly better opportunities to interact with and manage the data that would not have been possible with a 9600 baud serial line. We could write software in the SAIL language, which meant we would have much more expressive power than we had with the PDP8e or PDP11 computers and could apply more advanced algorithms. This made a real difference in the productivity in analyzing real data with powerful algorithms.
...
Later, the DECsystem-2020 was connected to the RTPP/PDP8e via a DMA interface. A DMA device on the PDP8e was connected by cable to a DMA interface (DR11-W) on the DECsystem-2020. An advantage of the DECsystem-2020 over the earlier PDP10 computer was that although it was a 36-bit/word computer, the 2020 could use less expensive 16-bit Unibus peripherals. Dan Kilgore of the Large Systems Group of DEC was contracted to write a TOPS-10 device driver to access this interface on the 2020. The 2020 system had the computational power required for larger, more complex projects because of its larger programming memory (512K 36-bit words), powerful instruction set, and the high-level SAIL language (Stanford Artificial Intelligence Language), an extended ALGOL-60 dialect. Later, Mort Schultz built a 500-Kbits/second high-speed serial line between the PDP8e and the DECsystem-2020 to control the PDP8e and thus BMON2 from the DECsystem-2020. This allowed the 2020 to treat the PDP8e as a slave processor and be controlled by software rather than typing on the PDP8e's terminal.
| Anchor | ||||
|---|---|---|---|---|
|
In addition to the RTPP, we acquired Comtal image processor systems that had Q-bus type PDP11 interfaces. These in turn were interfaced to a PDP11/40 computer that was connected to the DECsystem-2020 via software called SPIDER, a virtual device driver network. SPIDER allowed PDP11 computers to be accessed from the DECsystem2020 without writing a new DECsystem-2020 driver for each new PDP11 device. Bruce Shapiro, our expert on PDP11s, wrote a time-shared packet switcher on the PDP11/40 to connect PDP11 devices to this network. I wrote the device driver on the DECsystem-2020 to access these devices and make them available for DECsystem-2020 application software. Images acquired using the RTPP could be analyzed on the Comtals; Bruce used this to help analyze his nucleic acid electron micrographs.
| Anchor | ||||
|---|---|---|---|---|
|
Many software analysis systems were developed using the RTPP, especially in the area of 2D gels with the GELLAB-I system [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,24, 25, 26, 27, 28, 29, 30, 31, 32], a 2D gel exploratory data analysis system integrating the image-processing with statistical databases for multiple samples (myself); and RNA electron micrographs of secondary structure [10, 11, 32, 39, 48, 49] (Bruce Shapiro). After the RTPP was decommissioned, GELLAB-I was redeveloped as a portable software system using Unix/C/X-windows and was called GELLAB-II [42, 43, 44, 45] (see Lemkin's History of GELLAB for more details, references, and history of GELLAB-II). Much of the work with GELLAB-I and GELLAB-II in exploratory data analysis led to its application to the DNA microarray domain (see http://maexplorer.sourceforge.net/ ) MAExplorer []. A third-generation instantiation of this data-mining system is part of the Open2Dprot open-source project at http://open2dprot.sourceforge.net/ with the goal of extending proteomics data mining to 2D LC-MS, protein-arrays. Bruce went on to develop other RNA analysis software [35, 39, 47, 48, 49, 50], leading to the StructureLab project [50] and related RNA structure analysis (see his RNA structure research group ).
6.
| Anchor | ||||
|---|---|---|---|---|
|
Some of the design details were unique to the Real Time Picture Processor at that era of computer designs. A few of these are illustrated in the following figures. The design is explained in more detail in references [3, 4] and in technical reports [TR-7, TR-7a, TR-23]. Figures 4 and 5 show block diagrams of the components of the system. Figures 6 and 7 show the interactive control desk that the operator used to interact with the PDP8e and thus the RTPP.
| Anchor | ||||
|---|---|---|---|---|
|
Figure 4. The Real Time Picture Processor (RTPP) block diagram (reproduced with permission, from J. Histochem. Cytochem. [4], 1974). This shows additional parts of the system including a PDP11/20 message switcher to a PDP-10 Artificial Intelligence system PRDL (PRocedural Description Language) [5, TR-15] originally being developed on NIH's DCRT (now CIT) PDP-10 facility. The early microscope also had a 1024x1024 8-bit galvanometer scanner that could be used in place of the Quantimet 720 scanner. The later microscope was built around a Zeiss Axiomat microscope. An early high-quality grayscale display (Dicomed 31) was also used to make high-quality display images. Its functionality was replaced by the Quantimet grayscale buffer-memory display. The PDP8e accessed the RTPP using the BMON2 software [40, TR-21b]. The PDP-10 multiprocessor KL-10 system was a shared time-share computer at DCRT (now CIT). This was replaced in our design by a dedicated DECsystem-2020 when it became more cost-effective to have a dedicated computer. The DECsystem-2020 was a new microcoded processor that DEC was able to build for a fraction of the cost of the PDP-10. The PRDL [TR-15] and PROC10 [TR-8] image processing software were created to interface with the RTPP. We had considered creating a MAINSAIL(R) compiler for use with the GPPASM (GPP assembler program) [TR-16] so that we could program the GPP in a SAIL-like language only available on large PDP-10 class systems. Later, a light box for films was used with the Quantimet vidicon scanner (see Figure 18) with changeable 35mm lenses (not shown in this block diagram - see Figure 5) to scan autoradiograph and wet 2D gels, RNA electron micrographs, and other transparencies.
Anchor RTPP-Quantimet-PDP8e-blockDiagram RTPP-Quantimet-PDP8e-blockDiagram
Figure 5. A simplified block diagram of the Real Time Picture Processor illustrating the two types of input and the microscope control from BMON2 paper [40], 1980. (Reprinted from Computer Programs in Biomedicine, vol 11, Lemkin P., Lipkin, L., BMON2 - A distributed monitor system for biological image processing, pp 21-42, Copyright (1980), with permission from Elsevier.) The PDP8e computer directed the microscope stage to positions determined either manually by the operator or by a list of positions defined by the user and then controlled by the computer. Images could be acquired by the buffer memories for processing by the BMON2 system. Raw images as well as processed images could be displayed on the Quantimet 720 CRT display. TV camera input was from either of the two TV cameras that were easily changed. The user interacted with the hardware using the control panel connected to the PDP8e using the BMON2 image processing software system.
Anchor ControlDesk ControlDesk
Figure 6. RTPP Control Console was interfaced to the PDP8e and used to interact with the RTPP using the BMON2 buffer memory monitor operating system [40, TR-21, TR-21b]. See Figure 2 in [TR-21b] for the full description. It had various knobs (connected to A/D converters read by the PDP8e), lights for feedback, command buttons, toggle switches, and momentary toggle switches. Only some of these controls were used in the various programs, but having a variety of control options providing flexibility in the user interface. However, this was sometimes at the cost of added complexity and sometimes users had difficulty in learning the system because of this. ("All those knobs, buttons and switches!") However, this flexibility gave us the option of experimenting with various interaction modes that could then be optimized for particular analysis programs. This was before the computer mouse and graphical user interfaces became commonly available. (Click on this figure to bring up the high-resolution version of the figure. You may have to make your browser window larger.)
Anchor PhotoRTPPandConsole PhotoRTPPandConsole
Figure 6. RTPP Control Console was interfaced to the PDP8e and used to interact with 7. Photograph of the Quantimet-TV and control-console for the RTPP using the BMON2 buffer memory monitor operating system software [40, TR-21, TR-21b]. See Figure 2 in [TR-21b] for the full description. It had various knobs (connected to A/D converters read by the PDP8e), lights for feedback, command buttons, toggle switches, and momentary toggle switches. Only some of these controls were used in the various programs, but having a variety of control options providing flexibility in the user interface. However, this was sometimes at the cost of added complexity and sometimes users had difficulty in learning the system because of this. ("All those knobs, buttons and switches!") However, this flexibility gave us the option of experimenting with various interaction modes that could then be optimized for particular analysis programs. This was before the computer mouse and graphical user interfaces became commonly available. (Click on this figure to bring up the high-resolution version of the figure. You may have to make your browser window larger.)Figure 7. Photograph of the Quantimet-TV and control-console for the RTPP using the BMON2 software [40, TR-21, TR-21b, TR-23]. This was , TR-23]. This was taken after we had moved the RTPP to the Park Building in Rockville, MD. (Reproduced from a figure with permission from Environmental Health Perspectives, 1980 []). The control desk had a microscope joystick (X,Y) and Z-axis (focus) control; knobs (connected to A/D converters read by the PDP8e), switches and lights that could be configured in various ways by the BMON2 software. The small box shown in the lower left allowed us to control the 4 edges of a frame as (X,Y) positions. It used four bi-directional two-level spring-loaded switches in a (North, South, East, or West) configuration. These switches came from the LINC-8 and were perfect for this type of control. This allowed us to easily control the direction of a cursor - much as is done today using the mouse, which did not exist at the time). Real-time video control was performed using the RTPP buffer memory controller hardware, which in turn was configured by the PDP8e. The control desk gave us a lot of flexibility - even if its complexity was sometimes compared to that of the starship Enterprise. Various programs (BMON2, FLICKER [13], LANDMARK in the GELLAB-I system [15, 17, 31], and others) could use that subset of the controls most appropriate for the particular application.
| Anchor | ||||
|---|---|---|---|---|
|
The buffer memory image cards
...
During this time, we had the conviction, led by Lew Lipkin and George Carman, that anything that we wanted to be do in software could be done by a series of sequential gates. These could be proved Boolean algebraically correct using Karnaugh Maps , hardware finite state machines, and related techniques. George had just taken a microprogramming design course as part of his masters degree in computer hardware architecture and the design of buffer memories and the General Picture Processor (GPP) were perfect test beds in which to try out these new design principles which were relatively new for projects like this. Some of the design diagrams are shown in Figures 11 through 14 (from the Carman [4] paper). Figure 15 shows some examples of GPP microprogrammed instructions for manipulating the buffer memory data. The design was further described in some of the technical reports [TR-7, TR-7a, TR-16, TR-21, TR-21b, TR-22] listed at the end of this history. Because we were prototyping the system, the card was constructed using wire wrapping rather than multilayer printed circuit boards. A commercial version would have used printed circuit boards, but would only have been economically feasible if many copies of the RTPP were produced. Using complex multi-level printed circuit boards is generally too expensive for a research lab.
| Anchor | ||||
|---|---|---|---|---|
|
Figure 8. Front of a buffer memory image card containing 64Kb x 16-bits of dynamic RAM constructed from 4Kx1 bit dynamic RAM chips (Texas Instruments part number TMS 4030) initially sold by Texas Instruments and later second-sourced by National Semiconductor and Signetics. Four boards constituted a 256x256x16-bit pixel sub-image. Either the high or low 8-bit byte (or neither) could be displayed. Each 256x256 sub-image could be positioned to any part of the 860x720 pixel TV screen. For many applications, to create a 512x512 image, four 256x256 sub-images were grouped to form a 512x512 image.Anchor TMS4030chip TMS4030chip
Figure 9. The 4Kx1 bit dynamic RAM or DRAM chip (part number TMS 4030) initially sold by Texas Instruments and second sourced by National Semiconductor, and Signetics (shown here). These were the first affordable (about $20 at the time) DRAMs available in large quantitities. Because chip vendors want to assure customers that the parts will always be available, they license other chip makers to "second source" interchangeable chips. Our memory boards are a mix of the black, silver, and gold colored chips because we used several vendors.
Figure 10. Back of a buffer memory image card containing 64Kb x 16-bits of dynamic RAM constructed from 4Kx1 bit dynamic RAM (part number TMS 4030) chips initially sold by Texas Instruments and second sourced by National Semiconductor and Signetics. There were over 3,000 wirewraps on each board. The initial card was designed and hand wired by George Carman, and Cambion Corporation replicated 63 additional cards with wirewrap wiring lists generated by George. Their automatic robots would position the board for each of the wrap positions and then wire that point. The photograph illustrates how easy it is to get lost in this forest of pins and wires. Doing this by hand would have been impossible. George felt that no other company could build the boards in the time frame with the essential quality control we required. He was right. Only one of the 63 the boards delivered was defective, which was amazing considering the complexity and number of boards. Because of the high frequency signals involved, George put small black decoupling capacitors on each board to "tune" it to minimize cross talk. So each board, then, was in some sense unique.
Figure 11. The RTPP buffer memory control logic (reproduced with permission from J. Histochem. Cytochem. [4], 1974). "Each buffer memory is an asynchronous device that received I/O requests either from the general picture process (GPP) or the Quantimet for input or output. Given a request and an address, it first checked to see whether the last (high order 14-bit address) four-word buffer accessed was the same as that for the current request. If so, it did not have to do another memory (RAM) cycle and the signal OLDBWB signal is 'true'. When a read cycle occurred and a different FWB was needed, it checked to see if the FWB was 'dirtied', in which case it must write it back into the memory before the next current request could be proceed. Being dynamic RAMs, they must be refreshed (logic not shown) so as not to lose the data."Anchor Fig-RTPP-BufferMemory-control-logic Fig-RTPP-BufferMemory-control-logic
| Anchor | ||||
|---|---|---|---|---|
|
Figure 13. The RTPP triple line buffer logic (reproduced with permission from J. Histochem. Cytochem. [4], 1974). "The RTPP was designed to do 3x3 neighborhood image-processing in parallel in the GPP. Associated with the three lines is an effective Y dynamic address used to order the three lines as to (Y-1,Y,Y+1). In reading a raster line pattern into the triple line buffer, the oldest line must be replaced with the newest line. Similarly, the other two lines need to be adjusted as (Y-1,Y,Y+1) to (Y,Y+1,Y+2). By selecting the effective line address with a modulo three dynamic Y address counter, a dynamic Y address can be implemented. This is similar to the concept of buffer. The three X dynamic address vectors point to 3x3 neighborhood arrays in the line buffer. This neighborhood is called the current neighborhood. All of the dynamic address vectors are easily and efficiently programmed in the GPP to tessellate the current neighborhood along the three lines in the line buffer."
| Anchor | ||||
|---|---|---|---|---|
|
TR-7,TR-22])."
Figure 15. The Examples of RTPP instructions for the GPP (reproduced with permission from J. Histochem. Cytochem. [4], 1974). The Pi refers to a 3x3 pixel neighborhood that would be tessellated through the entire image. The GPP instructions [A HREF="#TR-22">TR-22] could be compiled by the GPPASM [A HREF="#TR-16">TR-16] assembler program running on the PDP8e and then loaded into the GPP instruction memory. A debugger for the GPP was DDTG that ran on the PDP8e [A HREF="#TR-2">TR-2] but controlled the GPP and buffer memories. We had also been evaluating collaborating on the construction of a MAINSAIL(R) Anchor Fig-RTPP-Examples-of-GPASM-code Fig-RTPP-Examples-of-GPASM-code compiler to generate GPP assembly code so we could program the RTPP in a SAIL-like language.
| Anchor | ||||
|---|---|---|---|---|
|
We describe a few of the main applications that used the Real Time Picture Processor to give a little flavor of its utility. Other projects are referred to in some of the lists of journal articles, technical reports, and in the list of RTPP users towards the end of this article.
| Anchor | ||||
|---|---|---|---|---|
|
Lewis Lipkin was working on the cellular effects of asbestos fibers on the induction of pleural sarcoma [10]. Lew developed a culture system that used a macrophage-like P388D1 tissue cell line to study the effect of asbestos fibers on cells. Asbestos fibers were cytotoxic to the P388D1 macrophages in tissue cultures. A microscope system was used to take photographs of samples over multiple days to study fiber-induced cytotoxicity for a range of asbestos and related fibers as in Figure 16. On incubation, the colonies lost numbers of cells, and giant cells occurred in places. In addition, significant changes occurred in cell morphology. Marta Wade, the technician who ran the cell-lines, shot time-lapse photographs of the cells with various types of asbestos fibers. This type of data was among the first candidates for use with the RTPP image-capture system using both phase contrast and differential interference optics on the Zeiss Axiomat microscope. We also used this tissue culture system to when analyzing protein differences of these samples with 2D gels [9]. One of the offshoots of this work was to measure the uniformity of cell-boundary adherence using hundreds of images gathered in 15-second intervals. These were analyzed with the boundary trace transform (BTT) [8, 9] illustrated in Figure 17.
| Anchor | ||||
|---|---|---|---|---|
|
Another of the projects Lew and I were also working on was the construction of a 3D brain atlas of an anolus lizard brain using aligned microtomed serial-section images. (Some of the goals for that project were similar to that of the National Library of Medicine's Visible Human. However, this was well before adequate computer technology and resources were available to implement such an atlas.) We wrote a PDP8e program to move the optical microscope stage in (X,Y) while keeping the center of the brain in the scanned visual center of the image field as we switched slides in a series of microtomed serial sections (the Z-axis control was not used in this operation). Each centered image could then be captured and saved as a disk file. The next section visible on the TV camera was compared with the image previously captured and stored in the image buffer memory. This procedure was iteratively repeated with subsequent slides to capture a sequence of aligned serial sections from a set of slides. The buffer memory data was saved at each point on 9-track magnetic tape. (This alignment software subsequently led to our involvement with 2D electrophoretic gels analysis [discussed below].) The set of serial sections could then be re-accessed sequentially to step through the brain slices centered at the selected point. We could generate movie loops with the BMON2 software to move virtually up and down through the serial sections at various frame rates. Because we could store sixteen 256x256 images in the buffer memory, the loops could have up to 16 sections and allowed us to visualize the 3D structure of part of the brain.
| Anchor | ||||
|---|---|---|---|---|
|
...
Figure 17. (Left) Boundary trace transform (BTT) of 233 images of 15-second interval scans of a single living P388D1 macrophage-like cell (left) (reproduced with permission from Environmental Health Perspectives, 1980 [9]). The boundaries for this set of images were traced by hand using a graphics tablet connected to the RTPP. The BTT is a 2D boundary frequency histogram where darker (higher frequency) pixels indicate boundaries of the cell that are more adherent to the glass slide and have less motility. BMON2 captured the set of images data and double buffered them to 9-track magnetic tape, with a 15-second image-sampling interval between scans. (Right) Illustration of a single cell captured by the RTPP using differential interference optics. The algorithm is described in several papers [8, 9]. Movies were made on the RTPP of a subsequence of 16 of the images in which you could see the cell trying to ingest the fiber, the fiber breaking through the other side of the cell (membrane), and then the cell backing up to try to reingest that part of the fiber, etc. These images allowed us to create a probability distribution of adhesion strength for boundary points of amoeboid cells. Points with low probability would flutter and extend. Looking at individual boundaries would not reveal this type of information; it was the ability to integrate large quantities of data that allowed these patterns to be detected. This was the way we thought the system should have been used - finding interesting results when no other method for doing so was possible.
| Anchor | ||||
|---|---|---|---|---|
|
In about 1977, Carl Merril of the National Institutes of Mental Health (NIMH) had a problem. He was working with 2-dimensional (2D) gel electrophoresis to determine protein shifts in E.coli amber amber mutant cultures. However, just overlaying the gels on a light box did not show the differences very well, and he knew a difference should show up in the (MW, pIe) range of the gels. We had constructed a rudimentary flicker comparison program for acquiring aligned serial sections (describe above). A mutual friend, who played ping-pong with Carl, suggested that he contact us because we had this new system to do image processing. We jury-rigged the RTPP with a vidicon camera that could scan the 2D gel image using a 35 mm camera lens using the program we had developed to flicker align the anolus brain. The program kept the image just scanned in the buffer memory and the other was in the active video. By alternating one image against the other (i.e., flickering, similar to what we did in aligning anolus brain serial sections), we were able to immediately see the amber mutant proteins. Spots had shifted across to the other side of the gel, which is why their detection eluded simple observation. Part of this observation discovered in the gel differences was later validated in wet-lab experiments. At that point, we started developing a series of programs starting with the original FLICKER program reported in [13] and later leading to the GELLAB-I system (i.e., GEL LABoratory for exploratory data analysis).
The FLICKER software allowed users to measure spots as a set of (X,Y) image positions and integrated-densities for each sample. We manually recorded this data for later analysis, manually defining spots to be corresponding if they aligned well with FLICKER.
Figure 18. This was the alternative image acquisition setup. (Reproduced from a figure with permission from a reprint from Environmental Health Perspectives, 1980 [9].) A high-quality uniform illumination Aristo light box and Quantimet Vidicon TV camera were used for non-microscope images. A gel autoradiograph, wet 2D gel, electron micrograph, or other transparent sample object was placed on the light box and scanned using the RTPP/Vidicon camera. A NBS standard neutral density step wedge was placed on the bottom of the scan area. This was captured along with the image and was used to calibrate the grayscale pixel data to optical density using a piecewise linear curve fitting algorithm (see [9] for an example). As an aside, the electronics workbench area in the background was where many of the local RTPP electronics assembly was performed in the Park Building.Anchor lightBoxCameraStation lightBoxCameraStation
...
Peter Sonderegger, while a post-doc at the National Institute of Child Health and Development (NICHD), used the GELLAB-I system with the RTPP to investigate how the expression of axonal proteins of sensory and motor neurons was influenced by non-neuronal cells [27, 29, 45]. At the same time we were investigating the feasibility of porting GELLAB-I (written in SAIL) to the PASCAL computer language. The inflexibility of PASCAL eventually led us to convert GELLAB-I to the portable C/UNIX/X-windows environment called GELLAB-II [42, 43, 44, 45] The DECsystem-10 SAIL version, GELLAB-I, was exported to research labs at Univ. of Chicago (Eric Lester) and Univ. of Kiel (Heinz Busse). The Unix version, GELLAB-II, was exported to a number of research labs around the world (CDC with Jim Myrick [44], Univ. Zurich with Peter Sonderegger, Agr. Univ. Norway with Trygve Krekling, and others) and led to a commercial subset version for Windows PCs called GELLAB-II++ by CSPI/Scanalytics.
| Anchor | ||||
|---|---|---|---|---|
|
.| Anchor | ||||
|---|---|---|---|---|
|
In the mid 1970s, Jacob Maizel (who later became chief of the Laboratory of Experimental and Computational Biology) visited the Image Processing Unit (IPU) carrying a set of electron micrographs of RNA molecules. While in the National Institute of Child Health and Development (NICHD), Jake had been using an early Hewlett Packard computer to manually trace the RNA molecules in the electron micrographs to determine the repetitive nature of features that were visible in these images. The traces, called secondary structure maps, indicated where double- and single-stranded regions occurred in the RNA. Knowledge of these regions is important for understanding RNA folding, which in turn is related to RNA's function. This manual task was tedious and Jake wondered whether the image processing hardware and software associated with the RTPP could be used to help generate secondary structure maps automatically. The RTPP was used to scan and preprocess some of these electron micrograph images [33, 34] that were input for this type of analysis. At the same time, Bruce had developed the circle transform [35] for biological shape description for describing cell mitosis with Lew and had published with Dr. Jack Sklansky, Ph.D. [37, 38], who was doing a sabbatical in the IPU at the time. Bruce became interested in the RNA folding problem after working on these shape descriptors for the cellular image domain.
...
Partly as a result of this collaboration, Jake became more interested in computing, which led him to purchase one of the early DEC VAX computers for his lab. As Jake became more involved with computational techniques, he realized the general importance, as did Lew, of using computers in biology. Eventually, this interest culminated with the National Cancer Institute's purchase of a Cray XMP supercomputer, and the establishment of what is today the Advanced Biomedical Computer Center in Frederick. The Cray XMP, at that time, was the first supercomputer in the world solely dedicated to biomedical research. Thus, the impact of the RTPP continues to be felt in the biological computation sciences 30 years later.
| Anchor | ||||
|---|---|---|---|---|
|
Figure 20. One of the early RNA electron micrographs scanned with the vidicon/RTPP system (Jacob Maizel, Bruce Shapiro, and Lewis Lipkin) [33, 34, 39, 48, 49]. The sample was adenovirus type 2 messenger RNA. Bruce developed boundary segmenters and boundary shape descriptors that could map electron micrograph data to the secondary structure.
| Anchor | ||||
|---|---|---|---|---|
|
The Real Time Picture Processor (RTPP) project occurred during that window of time when electronics were providing giant leaps forward in the ability to analyze data stored or collected by medical researchers. Prior to this, it would have been very difficult to collect, let alone analyze, the vast amounts of data it is now possible to acquire with new digital technology. This phase encompassed a major shift from analog to digital technology. There was also a shift to use combined hardware and software systems as part of the solution to these problems. Subsequently, as computers became faster, the solutions shifted again from mostly hardware-focused systems to those more software-based systems using less expensive but much more powerful computers becoming available. We were a part of the new digital frontier, making new digital imaging technology available to biomedical collaborators and then collectively devised new ways of look at this data. The tight feedback loop between developers and users was critical for discovering how these new digital imaging technologies could be used.
...
The major contributing factor to the success of this project was the great management that Lew Lipkin exhibited. He assembled a group of skilled individuals, fostered communication, encouraged the sharing of information and credit between them with group ownership, and supported the group so that they could get the job done. He collected the right people and trusted them. Bruce Shapiro and I applied what we were learning about image processing and computer science at the University of Maryland to the RTPP design. George Carman brought what he had just learned for his masters in computer hardware architecture about using microprogramming coding to implement flexible hardware designs. Lew was unique in that he understood and could communicate in the jargon of each of these areas. Everyone cooperated, helping everyone else where they could - constantly encouraged by Lew Lipkin's leadership and intellectual curiosity. With this positive communication between us, we made great progress and helped lead the way in automated cytology. It was quite an exciting time.
| Anchor | ||||
|---|---|---|---|---|
|
This history was compiled by myself (Peter Lemkin) with interviews, recollections, and content from Lewis Lipkin, George Carman, Bruce Shapiro, and Morton Shultz and some of the users (Carl Merril, Peter Sonderegger, Eric Lester). This article could not have been done without everyone's input. Thanks to my wife, Ellen Burchill, for assistance with the editing; Michelle Lyons, the Museum Curator, for helping with accessioning and editing; and to Maritta Grau's Frederick Scientific Publications group for editing suggestions. Any errors of omission or commission in this history are the responsibility of myself and of time. Memories tend to fade as the years pass. We attempted to reconstruct the history as best we could using additional information in papers, files, notes, technical reports, and interviews. We thank the journals that have opened their archives to free public access to old papers.
Since many of the companies and products mentioned in this article have disappeared, we found historical links on the Internet that were useful for providing the context in which to understand this project. When the history was completed, it was donated to the Museum of NIH in the Office of NIH History (http://history.nih.gov/) for use as part of their permanent online exhibits. We also wish to thank them for their help. A buffer memory image board from the original RTPP, selected papers and technical reports illustrating the design were also donated as artifacts. Many of the papers are available as PDF links to the journals, and most of the technical reports are available as downloadable PDF files in the References section of this Web site. The Office of NIH History has permanent exhibits as well as access to these artifacts.
| Anchor | ||||
|---|---|---|---|---|
|
The references are divided into journal articles and technical reports. Where it is possible, we provide links to the PDFs for open access from some of the journals as well as downloadable PDFs for most of the technical reports. The original technical report numbering is used in this history.
| Anchor | ||||
|---|---|---|---|---|
|
Lipkin, L.E., Watt, W.C., Kirsch, R.A.: The analysis, synthesis, and description of biological images. Ann N Y Acad Sci. 128(3): 984-1012, 1966.Anchor Lipkin66 Lipkin66
Lipkin, L.E., Lemkin, P.F., Carman, G.: Automated autoradiographic grain counting in human determined context. J. Histochem. Cytochem. 22(7): 755-765, 1974. (PDFAnchor Lipkin74 Lipkin74 
)
Lemkin, P.F., Carman, G., Lipkin, L., Shapiro, B., Schultz, M., Kaiser, P.: A real time picture processor for use in biologic cell identification. I. System design. J. Histochem. Cytochem. 22(7): 725-731, 1974. (PDFAnchor RTPP-JHC-I-paper RTPP-JHC-I-paper )
Carman, G., Lemkin, P.F., Lipkin, L., Shapiro, B., Schultz, M., Kaiser, P.: A real time picture processor for use in biologic cell identification. II. Hardware implementation. J. Histochem. Cytochem. 22(7): 732-740, 1974. (PDFAnchor RTPP-JHC-II-paper RTPP-JHC-II-paper )
Shapiro, B., Lemkin, P.F., Lipkin, L.: The application of artificial intelligence techniques to biologic cell identification. J. Histochem. Cytochem. 22(7): 741-750, 1974. (PDFAnchor PRDL-ECR-paper PRDL-ECR-paper )
Schultz, M.L., Lipkin, L.E., Wade, M.J., Lemkin, P.F., Carman, G.M.: High resolution shading correction. J. Histochem. Cytochem.22(7): 751-754, 1974. (PDFAnchor Schultz74 Schultz74 )
Shapiro, H.M., Bryan, S.D., Lipkin, L.E., Stein, P.G., Lemkin, P.F.: Computer-aided microspectrophotometry of biological specimens. Exp Cell Res. 67(1): 81-89, 1971.Anchor Shapiro71 Shapiro71
Lemkin, P.F.: The boundary trace transform: An edge and region enhancement transform. Comp. Graphics Image Processing 9: 150-165, 1979.Anchor Lemkin79a Lemkin79a
Lemkin, P.F., Lipkin, L., Merril, C., Shiffrin, S.: Protein abnormalities in macrophages bearing asbestos. Environ. Health Perspect. 34: 5-89, 1980. (PDF)Anchor Lemkin80a Lemkin80a
Lipkin, L.E.: Cellular effects of asbestos and other fibers: correlations with in vivo induction of pleural sarcoma. Environ. Health Perspect. 34:91-102, 1980. (PDF)Anchor Lemkin80b Lemkin80b
Lemkin, P.F.: An approach to region splitting. Comp. Graphics Image Processing 10: 281-288, 1979.Anchor Lemkin79b Lemkin79b
Lemkin, P.F., Lipkin, L.: Use of the positive difference transform for RBC elimination in bone marrow smear images. Anal. Quant. Cytol. 1(1): 67-76, 1979.Anchor Lemkin79c Lemkin79c
Lemkin, P.F., Merril, C., Lipkin, L., Van Keuren, M., Oertel, W., Shapiro, B., Wade, M., Schultz, M., Smith, E.: Software aids for the analysis of 2D gel electrophoresis images. Comput. Biomed. Res. 12: 517-544, 1979.Anchor Lemkin79d Lemkin79d
Lester, E.P., Lemkin, P.F., Cooper, H.L., Lipkin, L.E.: Computer-assisted analysis of two-dimensional electrophoresis of human peripheral blood lymphocytes. Clin. Chem. 26: 1392-1402, 1980. (PDFAnchor Lemkin80c Lemkin80c )
Lipkin, L.E., Lemkin, P.F.: Data base techniques for multiple PAGE (2D gel) analysis. Clin. Chem. 26: 1403-1413, 1980. (PDFAnchor Lemkin80d Lemkin80d )
Lemkin, P.F., Lipkin, L.: GELLAB: A computer system for 2D gel electrophoresis analysis. I. Segmentation and preliminaries. Comput. Biomed. Res. 14: 272-297, 1981.Anchor Lemkin81a Lemkin81a
Lemkin, P.F., Lipkin, L.: GELLAB: A computer system for 2D gel electrophoresis analysis. II. Spot pairing. Comput. Biomed. Res. 14: 355-380, 1981.Anchor Lemkin81b Lemkin81b
Lemkin, P.F., Lipkin, L.: GELLAB: A computer system for 2D gel electrophoresis analysis. III. Multiple gel analysis. Comput. Biomed. Res. 14: 407-446, 1981.Anchor Lemkin81c Lemkin81c
Lester, E.P., Lemkin, P.F., Lipkin, L.E.: New dimensions in protein analysis - 2D gels coming of age through Image Processing. Anal. Chem. 53: 390A-397A, 1981. (PDF)Anchor Lemkin81d Lemkin81d
Lester, E.P., Lemkin, P.F., Lipkin, L.E., Cooper, H.L.: A two-dimensional electrophoretic analysis of protein synthesis in resting and growing lymphocytes in vitro. J. Immunol. 126: 1428-1434, 1981.Anchor Lemkin81e Lemkin81e
Lemkin, P.F., Lipkin, L.E., Lester, E.P.: Some extensions to the GELLAB 2D electrophoresis gel analysis system. Clin. Chem. 28: 840-849, 1982. (PDFAnchor Lemkin82a Lemkin82a )
Lester, E.P., Lemkin, P.F., Lipkin, L.E.: A two-dimensional gel analysis of autologous T and B lymphoblastoid cell lines. Clin. Chem.28: 828-839, 1982. (PDFAnchor Lester82a Lester82a )
Lester, E.P., Lemkin, P.F., Lowery, J.F., Lipkin, L.E.: Human leukemias: A preliminary 2D electrophoretic analysis. Electrophoresis 3: 364-375, 1982.Anchor Lester82b Lester82b
Howard, R.J., Aley, S.B., Lemkin, P.F.: High resolution comparison of Plasmodium Knowlesi clones of different variant antigen phenotypes by 2D gel electrophoresis and computer analysis. Electrophoresis 4: 420-427, 1983.Anchor Howard83 Howard83
Lemkin, P.F., Lipkin, L.E.: 2D Electrophoresis gel data base analysis: Aspects of data structures and search strategies in GELLAB. Electrophoresis 4: 71-81, 1983.Anchor Lemkin83a Lemkin83a
Lester, E.P., Lemkin, P.F., Lipkin, L.E.: States of differentiation in leukemias: A 2D gel analysis. In Rowley, J. D. and Ultmann, J. E. (Eds.): Proceedings of 5th Annual Bristol Myers Symposium on Cancer Research. Chromosomes and Cancer: From Molecules to Man. New York, Academic Press, 1983, pp. 226-245.Anchor Lester83a Lester83a
Lemkin, P.F., Sonderegger, P., Lipkin, L.: Identification of coordinate pairs of polypeptides: A technique for screening of putative precursor product pairs in 2D gels.Clin. Chem. 30: 1965-1971, 1984. (PDFAnchor Lemkin84a Lemkin84a )
Lester, E.P., Lemkin, P F., Lipkin, L.E.: Protein indexing in leukemias and lymphomas. Ann. N.Y. Acad. Sci. 428: 158-172, 1984.Anchor Lester84a Lester84a
Sonderegger, P., Lemkin, P.F., Lipkin, L., Nelson, P.: Differential modulation of the expression of axonal proteins by non-neuronal cells and the peripheral and central nervous system. EMBO J. 4: 1395-1401, 1985. (PDF)Anchor Sonderegger85 Sonderegger85
Lemkin, P.F., Lipkin, L.E.: GELLAB: Multiple 2D electrophoretic gel analysis. In Allen, R. and Arnaud (Eds.): Electrophoresis '81. New York, W. De Gruyter, 1981, pp. 401-411.Anchor Lemkin81f Lemkin81f
Lemkin, P.F. , Lipkin, L.E.: Database techniques for 2D electrophoretic gel analysis. In Geisow, M. and Barrett, A. (Eds.): Computing in Biological Science. North Holland, Elsevier, 1983, pp. 181-226.Anchor Lemkin83b Lemkin83b
Lester, E.P., Lemkin, P.F.: A 'GELLAB' computer assisted 2D gel analysis of states of differentiation in hematopoietic cells. In Neuhoff, V. (Ed.): Electrophoresis '84. Chemie, Springer-Verlag, 1984, pp. 309-311.Anchor Lester84a Lester84a
Lemkin, P.F., Shapiro, B., Lipkin, L., Maizel, J., Sklansky, J., Schultz, M.: Preprocessing of electron micrographs of nucleic acid molecules for automatic analysis by computer. II. Noise removal and gap filling. Comput. Biomed. Res. 12: 615-630, 1979.Anchor Lemkin79d Lemkin79d
Lipkin, L., Lemkin, P.F., Shapiro, B., Sklansky, J.: Preprocessing of electron micrographs of nucleic acid molecules for automatic analysis by computer. Comput. Biomed. Res. 12: 279-289, 1979.Anchor Lipkin79b Lipkin79b
Shapiro, B., Lipkin L.: The circle transform, an articulable shape descriptor. Comput. Biomed. Res. 10: 511-28, 1977.Anchor Shapiro77a Shapiro77a
Shapiro, B.: Language processor generation with BNF inputs: methods and implementation. Comp. Programs. Biomedicine 7:85-98, 1977.Anchor Shapiro77b Shapiro77b
Shapiro, B., Pisa, J., Sklansky, J.: Skeletons from sequential boundary data. Proc. Intl. Conf. On Pattern Recognition and Image Processing. IEEE Comp. Soc. Press, Los Angeles, CA., 265-270, 1979.Anchor Shapiro79a Shapiro79a
Shapiro, B., Pisa, J., Sklansky, J.: Skeleton generation from xy boundary sequences. Comp. Vision Graphics Image Processing 15(2) 136-153, 1981.Anchor Shapiro81a Shapiro81a
Shapiro, B.S., Lipkin, L.E., Maizel, J.V.: Computerized generation of secondary structure maps for nucleic acids. Comp. Biomed. Res.12(6):545-568, 1979.Anchor Shapiro79b Shapiro79b
Lemkin, P.F., Lipkin, L.: BMON2 - A distributed monitor system for biological image processing. Computer Programs in Biomedicine 11: 21-42, 1980. (PDF)Reprinted from COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE, Vol 11, Lemkin PD and LipkinL, BMON2 - A distributed monitor system for biological image processing, Pages 21-42, Copyright (1980), with permission from Elsevier.Anchor BMON2-CPIB-paper BMON2-CPIB-paper
Lipkin, B.S., Rosenfeld, A. (Eds): Picture Processing and Psychopictorics. Academic Press, New York, 1970, pps 526.Anchor Lipkin70 Lipkin70
Lemkin, P.F.: GELLAB-II: A workstation based 2D electrophoresis gel analysis system. In Endler, T. and Hanash, S. (Eds.): Proceedings of 2D Electrophoresis. West Germany, VCH Press, 1989, pp. 52-57. (This was the announcement of GELLAB-II)Anchor Lipkin89g Lipkin89g
Lemkin, P.F., Lester, E.P.: Database and search techniques for 2D gel protein data: A comparison of paradigms for exploratory data analysis and prospects for biological modeling. Electrophoresis 10(2): 122-140, 1989.Anchor Lipkin89h Lipkin89h
Robinson, M.K., Myrick, J.E., Henderson, L.O., Coles, C.D., Powell, M.K., Orr, G.A., Lemkin, P.F.: Two-dimensional protein electrophoresis and multiple hypothesis testing to detect potential serum protein biomarkers in children with fetal alcohol syndrome. Electrophoresis 16: 1176-1183, 1995.Anchor Robinson95 Robinson95
Stoeckli, E.T., Lemkin, P.F., Kuhn, T.B., Ruegg, M.A., Heller, M., Sonderegger, P.: Identification of proteins secreted from axons of embryonic dorsal-root-ganglia neurons. Eur. J. Biochem. 180: 249-258, 1989.Anchor Stoeckli89 Stoeckli89
Lemkin, P.F., Thornwall, G., Walton, K., Hennighausen, L: The Microarray Explorer tool for data mining of cDNA microarrays - application for the mammary gland, Nucleic Acids Res. 20(22): 4452-4459, 2000.Anchor Lemkin00a Lemkin00a
Shapiro, B.A.: An algorithm for comparing multiple RNA secondary structures. Comput. Appl. Biosci. 4(3): 387-393, 1988.Anchor Shapiro88 Shapiro88
Margalit, H., Shapiro, B.A., Oppenheim, A.B., Maizel, J.V. Jr.: Detection of common motifs in RNA secondary structures. Nucleic Acids Res. 17(12): 4829-4845, 1989.Anchor Margalit89 Margalit89
Le, S.Y., Owens, J., Nussinov, R., Chen, J.H., Shapiro, B., Maizel, J.V.: RNA secondary structures: comparison and determination of frequently recurring substructures by consensus. Comput. Appl. Biosci. 5(3): 205-210, 1989.Anchor Le89 Le89
Shapiro, B.A., Kasprzak, W.: STRUCTURELAB: a heterogeneous bioinformatics system for RNA structure analysis. J Mol. Graph.14(4): 194-205, 222-224, 1996.Anchor Shapiro96 Shapiro96
| Anchor | ||||
|---|---|---|---|---|
|
These reports were archived in the U.S. Department of Commerce's National Technical Information Service (NTIS) in Springfield VA. They currently offer online reports from 1990 on their web site http://www.ntis.gov/, although older reports can be ordered. Some additional reports appeared as University of Maryland Computer Science Center technical reports.
...
Overview
Content Tools
ThemeBuilder