...

Lew's solution was to view the slide as an array, a 2-dimensional (2D) matrix where each visible area had a unique 2-dimensional address on the slide. The sections were very thin so that all the grains at a location were visible; the Z-axis in this case could be ignored. Lew's system used a list of random number XY positions, which were applied to each slide. Dr. Vinichaichol would go to these areas and count whatever grains were there. If there were no cells, there were zero counts. And suddenly everything fell together. The new method was what Dr. Fitzgerald needed. This result was published in 1968 in the American Journal of PathologyFitzgerald, P. et.al., 53(6):953-970, "Pancreatic acinar cell regeneration. V. Analysis of variance of the autoradiographic labeling index (thymidine-H3)."

...

In 1968, I (Peter Lemkin) joined Lew's group to work on programming the LINC-8 along with Howard Shapiro of the PRB, and Russell Kirsch, Don Orser, and Phil Stein from the NBS who had been involved in the project. The first lab was in rental space in the Auburn Building across from the Bethesda Chevy Chase Rescue Squad where we would hear the fire trucks when they went out on a call. The group moved to the brand new Building 36 on the NIH Bethesda campus around 1970, which was a much better environment. (Building 36 was demolished in 2006.) The LINC-8 controlled a stepping-stage and a galvanometer scanner with a photomultiplier detector on a Leitz microscope, which was an early step in automated cytology [7]. It was very slow, but did offer high-quality 8-bit data. The problem was analysis power - in terms of scanning speed, CPU speed, image memory, analysis software, and analysis memory. It became clear that we did not have the hardware resources required to do complex image processing on the types of data we were determined to analyze. However, I learned to write hardware control software on the LINC-8 as it was truly a dedicated laboratory instrument computer ideal for connecting to laboratory equipment. This experience set the stage for the next generation of computer-controlled microscopes we tackled.

...

Scientists used the RTPP as finally constructed to analyze data in a variety of biomedical domains including optical microscope images of optical serial sections of brain tissue, stained bone marrow smears, and tissue cultures using phase contrast and differential interference optics. The latter was used in tracking cell membrane extents of macrophages in tissue culture over time as the cells tried to phagocytize various types of asbestos fibers. The goal was to better understand fiber carcinogenicity and the dynamics of fiber ingestion [8, 9, 10]. The bone marrow smear image analysis was part of my Ph.D. dissertation [11, 12, TR-653, TR-655]. The RTPP was also used for 2D electrophoretic gel images for a variety of biological materials [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32], and for RNA electron micrographs of secondary structure, which was part of Bruce Shapiro's Ph.D. dissertation [33, 34, 35, 36, 37, 38, TR-BAS78].

2. The Real Time Picture Processor Development Team

...

• Lewis Lipkin, M.D., (mathematics and physical chemistry, and a neuropathologist), Head of the Image Processing Section (IPS); previously the (PRB, NINDB) and then the Image Processing Unit (IPU) in the NCI.
• Peter Lemkin, Ph.D. & M.S. EE, computer scientist and electrical engineer, IPS/NCI, and previously in (PRB, NINDB) and in IPU/NCI
• George Carman, M.S. EE, electrical engineer and computer hardware architecture, Technical Development Section (TDS), NINDB; Carman Engineering (now Lucidyne Corp ).
• Morton Schultz, B.S. EE, electrical engineer, IPS/NCI, and previously in IPU/NCI
• Bruce Shapiro, Ph.D., B.S. math & physics, computer scientist, IPS/NCI, and previously in (PRB, NINDB),and in IPU/NCI
• Sprague Hazard, mechanical engineer (contractor consultant)
• Peter Kaiser, B.S. CS, computer scientist (IPU) in the NCI
• Earl Smith, M.S. CS, computer scientist (IPU) in the NCI
• Dan Kilgore, B.S. EE, computer programmer [Digital Equipment Corp  (DEC) software engineer]
• Tom Duval and later Jim Camper, electronics technicians - helped construct the RTPP racks, and power-supplies cabinets
• Cambion Corporation, wire-wrapped the remaining 63 buffer memory boards and the back-planes

Where: PRB was the Perinatal Research Branch in the National Institute of the Neurological Disease and Blindness (NINDB). IPU was the Image Processing Unit of the National Cancer Institute (NCI). The IPU later became the Image Processing Section (IPS).

Who: The initial participants were Dr. Lewis Lipkin, myself, and George Carman. Later in the process, Morton Schultz and Bruce Shapiro joined the design group. Peter Kaiser and Earl Smith participated for a few years. During this time, Bruce Shapiro and I were part-time Ph.D. students in the Computer Science Department of the University of Maryland with Professor Azriel Rosenfeld, one of the early leaders in the field of image processing. Both Bruce [35, 37, 38, 39, TR-BAS78] and I [8,12, TR-653, TR-655] wrote dissertations on image processing. We were able to combine some of our applied NIH research as part of our Ph.D. research and use what we were learning about image processing and computer science to our NIH research. George was able to apply many of the ideas he had learned with his masters in Computer Hardware Architecture.

...

• Lewis Lipkin: optical microscopy of serial brain sections and macrophage motility measurements with asbestos
• Peter Lemkin: bone marrow smear analysis, 2D gel electrophoresis
• Bruce Shapiro: RNA secondary structure of electron micrographs
• Carl Merril: NIMH/NIH - 2-dimensional (2D) gel electrophoresis, E.coli mutants and macrophages with asbestos
• Jacob Maizel: NICHD/NIH, with Bruce Shapiro - RNA electron microscopy of secondary structure
• Eric Lester: NCI, U. Chicago, and oncology practice - 2D gel electrophoresis on human leukemias
• Steve Aley and Russell Howard: NIAID/NIH - 2D gel electrophoresis of Plasmodium knowlesi clones
• Peter Wirth and Snorri Thorgeirsson: NCI/NIH - 2D gel electrophoresis on liver cell lines
• Peter Sonderegger: NICHD/NIH and U. Zurich - 2D gel electrophoresis of axonal proteins of sensory and motor neurons

...

Figure 4. The Real Time Picture Processor (RTPP) block diagram (reproduced with permission, from J. Histochem. Cytochem. [4], 1974). This shows additional parts of the system including a PDP11/20 message switcher to a PDP-10 Artificial Intelligence system PRDL (PRocedural Description Language) [5, TR-15] originally being developed on NIH's DCRT (now CIT) PDP-10 facility. The early microscope also had a 1024x1024 8-bit galvanometer scanner that could be used in place of the Quantimet 720 scanner. The later microscope was built around a Zeiss Axiomat microscope. An early high-quality grayscale display (Dicomed 31) was also used to make high-quality display images. Its functionality was replaced by the Quantimet grayscale buffer-memory display. The PDP8e accessed the RTPP using the BMON2 software [40, TR-21b]. The PDP-10 multiprocessor KL-10 system was a shared time-share computer at DCRT (now CIT). This was replaced in our design by a dedicated DECsystem-2020 when it became more cost-effective to have a dedicated computer. The DECsystem-2020 was a new microcoded processor that DEC was able to build for a fraction of the cost of the PDP-10. The PRDL [TR-15] and PROC10 [TR-8] image processing software were created to interface with the RTPP. We had considered creating a MAINSAIL(R)  compiler for use with the GPPASM (GPP assembler program) [TR-16] so that we could program the GPP in a SAIL-like language only available on large PDP-10 class systems. Later, a light box for films was used with the Quantimet vidicon scanner (see Figure 18) with changeable 35mm lenses (not shown in this block diagram - see Figure 5) to scan autoradiograph and wet 2D gels, RNA electron micrographs, and other transparencies.

...

Figure 5. A simplified block diagram of the Real Time Picture Processor illustrating the two types of input and the microscope control from BMON2 paper [40], 1980. (Reprinted from Computer Programs in Biomedicine, vol 11, Lemkin P., Lipkin, L., BMON2 - A distributed monitor system for biological image processing, pp 21-42, Copyright (1980), with permission from Elsevier.) The PDP8e computer directed the microscope stage to positions determined either manually by the operator or by a list of positions defined by the user and then controlled by the computer. Images could be acquired by the buffer memories for processing by the BMON2 system. Raw images as well as processed images could be displayed on the Quantimet 720 CRT display. TV camera input was from either of the two TV cameras that were easily changed. The user interacted with the hardware using the control panel connected to the PDP8e using the BMON2 image processing software system.

...

Figure 7. Photograph of the Quantimet-TV and control-console for the RTPP using the BMON2 software [40, TR-21, TR-21b, TR-23]. This was taken after we had moved the RTPP to the Park Building in Rockville, MD. (Reproduced from a figure with permission from Environmental Health Perspectives, 1980 []). The control desk had a microscope joystick (X,Y) and Z-axis (focus) control; knobs (connected to A/D converters read by the PDP8e), switches and lights that could be configured in various ways by the BMON2 software. The small box shown in the lower left allowed us to control the 4 edges of a frame as (X,Y) positions. It used four bi-directional two-level spring-loaded switches in a (North, South, East, or West) configuration. These switches came from the LINC-8 and were perfect for this type of control. This allowed us to easily control the direction of a cursor - much as is done today using the mouse, which did not exist at the time). Real-time video control was performed using the RTPP buffer memory controller hardware, which in turn was configured by the PDP8e. The control desk gave us a lot of flexibility - even if its complexity was sometimes compared to that of the starship Enterprise. Various programs (BMON2, FLICKER [13], LANDMARK in the GELLAB-I system [15, 17, 31], and others) could use that subset of the controls most appropriate for the particular application.

...

Figure 11. The RTPP buffer memory control logic (reproduced with permission from J. Histochem. Cytochem. [4], 1974). "Each buffer memory is an asynchronous device that received I/O requests either from the general picture process (GPP) or the Quantimet for input or output. Given a request and an address, it first checked to see whether the last (high order 14-bit address) four-word buffer accessed was the same as that for the current request. If so, it did not have to do another memory (RAM) cycle and the signal OLDBWB signal is 'true'. When a read cycle occurred and a different FWB was needed, it checked to see if the FWB was 'dirtied', in which case it must write it back into the memory before the next current request could be proceed. Being dynamic RAMs, they must be refreshed (logic not shown) so as not to lose the data."

 Figure 12. The RTPP general picture processor (GPP) bus structure (reproduced with permission from J. Histochem. Cytochem. [4], 1974). "The instruction addressing sequence is done serially. That is, Pi is addressed, then P2, then P3. Let 'c(.)' denote 'contents of current memory location'. If any address is immediate, no memory fetch is done. Rather, the PM data, c(P), is enabled onto the data bus DB. If direct addressing mode c(c(P)) is used, the PM data c(P) is enabled onto the data address bus, DAB, and then loaded onto the appropriate data field address register. The memory then enables its data, c(c(P)), onto the data base, DB. If indirect mode is used then the same sequence is repeated as for direct mode, but c(c(P)) is enabled back onto the DAB instead of the DB. Then the data address field register addressed is loaded and the c(c(c(P))) from that memory is enabled onto the DB. A conflict may occur in the use of the indirect mode from the 'MOVE' instruction. This is resolved by storing the source data in the data bus register, DBR, temporarily. Various devices and memories are connected to the bus structure and interact when the control section activates them. The average GPP instruction time is designed to be on the order of 250 nanoseconds." (Click on this figure to bring up the high-resolution version of the figure. You may have to make your browser window larger.)

...

Figure 13. The RTPP triple line buffer logic (reproduced with permission from J. Histochem. Cytochem. [4], 1974). "The RTPP was designed to do 3x3 neighborhood image-processing in parallel in the GPP. Associated with the three lines is an effective Y dynamic address used to order the three lines as to (Y-1,Y,Y+1). In reading a raster line pattern into the triple line buffer, the oldest line must be replaced with the newest line. Similarly, the other two lines need to be adjusted as (Y-1,Y,Y+1) to (Y,Y+1,Y+2). By selecting the effective line address with a modulo three dynamic Y address counter, a dynamic Y address can be implemented. This is similar to the concept of buffer. The three X dynamic address vectors point to 3x3 neighborhood arrays in the line buffer. This neighborhood is called the current neighborhood. All of the dynamic address vectors are easily and efficiently programmed in the GPP to tessellate the current neighborhood along the three lines in the line buffer."

...

Figure 14. The RTPP GPP control logic finite state machine (reproduced with permission from J. Histochem. Cytochem. [4], 1974). "The GPP control logic is implemented as a finite state machine where the states of the system are defined from the logic flow of the system. This consists of the various bus, register enable and load signals. The state is a function of the current state and the current operator. Thus, to extend the machine, additional states and transitions between states would have been added (see [TR-7,TR-22])."

...

Figure 15. The Examples of RTPP instructions for the GPP (reproduced with permission from J. Histochem. Cytochem. [4], 1974). The Pi refers to a 3x3 pixel neighborhood that would be tessellated through the entire image. The GPP instructions [A HREF="#TR-22">TR-22] could be compiled by the GPPASM [A HREF="#TR-16">TR-16] assembler program running on the PDP8e and then loaded into the GPP instruction memory. A debugger for the GPP was DDTG that ran on the PDP8e [A HREF="#TR-2">TR-2] but controlled the GPP and buffer memories. We had also been evaluating collaborating on the construction of a MAINSAIL(R)  compiler to generate GPP assembly code so we could program the RTPP in a SAIL-like language.

...

Another of the projects Lew and I were also working on was the construction of a 3D brain atlas of an anolus lizard brain using aligned microtomed serial-section images. (Some of the goals for that project were similar to that of the National Library of Medicine's Visible Human. However, this was well before adequate computer technology and resources were available to implement such an atlas.) We wrote a PDP8e program to move the optical microscope stage in (X,Y) while keeping the center of the brain in the scanned visual center of the image field as we switched slides in a series of microtomed serial sections (the Z-axis control was not used in this operation). Each centered image could then be captured and saved as a disk file. The next section visible on the TV camera was compared with the image previously captured and stored in the image buffer memory. This procedure was iteratively repeated with subsequent slides to capture a sequence of aligned serial sections from a set of slides. The buffer memory data was saved at each point on 9-track magnetic tape. (This alignment software subsequently led to our involvement with 2D electrophoretic gels analysis [discussed below].) The set of serial sections could then be re-accessed sequentially to step through the brain slices centered at the selected point. We could generate movie loops with the BMON2 software to move virtually up and down through the serial sections at various frame rates. Because we could store sixteen 256x256 images in the buffer memory, the loops could have up to 16 sections and allowed us to visualize the 3D structure of part of the brain.

...

Figure 16. Photomicrograph of P388D1 macrophage-like cells 24 hours after amosite asbestos treatment used to study fiber-induced cytotoxicity [9] (reproduced with permission from Environmental Health Perspectives, 1980).

Figure 17. (Left) Boundary trace transform (BTT) of 233 images of 15-second interval scans of a single living P388D1 macrophage-like cell (left) (reproduced with permission from Environmental Health Perspectives, 1980 [9]). The boundaries for this set of images were traced by hand using a graphics tablet connected to the RTPP. The BTT is a 2D boundary frequency histogram where darker (higher frequency) pixels indicate boundaries of the cell that are more adherent to the glass slide and have less motility. BMON2 captured the set of images data and double buffered them to 9-track magnetic tape, with a 15-second image-sampling interval between scans. (Right) Illustration of a single cell captured by the RTPP using differential interference optics. The algorithm is described in several papers [8, 9]. Movies were made on the RTPP of a subsequence of 16 of the images in which you could see the cell trying to ingest the fiber, the fiber breaking through the other side of the cell (membrane), and then the cell backing up to try to reingest that part of the fiber, etc. These images allowed us to create a probability distribution of adhesion strength for boundary points of amoeboid cells. Points with low probability would flutter and extend. Looking at individual boundaries would not reveal this type of information; it was the ability to integrate large quantities of data that allowed these patterns to be detected. This was the way we thought the system should have been used - finding interesting results when no other method for doing so was possible.

...

In about 1977, Carl Merril of the National Institutes of Mental Health (NIMH) had a problem. He was working with 2-dimensional (2D) gel electrophoresis to determine protein shifts in E.coli amber mutant cultures. However, just overlaying the gels on a light box did not show the differences very well, and he knew a difference should show up in the (MW, pIe) range of the gels. We had constructed a rudimentary flicker comparison program for acquiring aligned serial sections (describe above). A mutual friend, who played ping-pong with Carl, suggested that he contact us because we had this new system to do image processing. We jury-rigged the RTPP with a vidicon camera that could scan the 2D gel image using a 35 mm camera lens using the program we had developed to flicker align the anolus brain. The program kept the image just scanned in the buffer memory and the other was in the active video. By alternating one image against the other (i.e., flickering, similar to what we did in aligning anolus brain serial sections), we were able to immediately see the amber mutant proteins. Spots had shifted across to the other side of the gel, which is why their detection eluded simple observation. Part of this observation discovered in the gel differences was later validated in wet-lab experiments. At that point, we started developing a series of programs starting with the original FLICKER program reported in [13] and later leading to the GELLAB-I system (i.e., GEL LABoratory for exploratory data analysis).

...

Figure 18. This was the alternative image acquisition setup. (Reproduced from a figure with permission from a reprint from Environmental Health Perspectives, 1980 [9].) A high-quality uniform illumination Aristo light box and Quantimet Vidicon TV camera were used for non-microscope images. A gel autoradiograph, wet 2D gel, electron micrograph, or other transparent sample object was placed on the light box and scanned using the RTPP/Vidicon camera. A NBS standard neutral density step wedge was placed on the bottom of the scan area. This was captured along with the image and was used to calibrate the grayscale pixel data to optical density using a piecewise linear curve fitting algorithm (see [9] for an example). As an aside, the electronics workbench area in the background was where many of the local RTPP electronics assembly was performed in the Park Building.

...