Photogrammetry and 3-D Assembly

Three-dimensional (3-D) assembly refers to the use of high-accuracy, in-place, 3-D coordinate measurement devices for the digital assembly of parts. This process is often referred to as computer-aided manufacturing (CAM) or gaugeless manufacturing. Whatever the name, 3-D assembly is replacing classical techniques centered on the use of tools, gauges and other mechanical processes of part assembly. In a nutshell, 3-D assembly can produce more accurate assemblies more rapidly and at lower cost.

ADVERTISEMENT

In the aerospace industry, where parts are large and required accuracy is high, and where the building and maintenance of tooling and gauges is very expensive, 3-D assembly is gaining much wider adoption. It is also well-suited to shipbuilding applications where, although accuracy is less of a concern, the scale of the parts being assembled is very large, indeed often too large for gauges. In the automotive industry, 3-D assembly is utilized in many areas, for example, the dimensional inspection of incoming parts to ensure good fit-up during later assembly, and for in-line process monitoring and control.

Although not as widely used as 3-D assembly technologies such as laser trackers and articulating arms, photogrammetry has been found by manufacturers to be a very versatile and reliable 3-D assembly tool, which offers a unique set of capabilities, especially in situations and environments where alternative approaches fall short, either from the standpoints of accuracy and reliability, or practicability.

Photogrammetry, as the name implies stands for “photographic metrology.” The technique works in much the same way as human stereo vision, where your eyes perceive depth partially as a function of the angle subtended by the two intersecting light rays running from the point of interest to your two eyes. The difference with photogrammetry, however, is that there is no limit to the number of images used in the determination of 3-D position. Each 2-D image that “sees” a point of interest contributes a light ray in space, and the intersection of all corresponding rays yields the required XYZ coordinates for the point via a mathematical reconstruction of 3-D shape from the multiple 2-D images. The basic process is illustrated in Figure 1.

Figure 1: The photogrammetric process: An object is photographed in multiple 2-D images which are then used to digitally reconstruct the 3-D object.

One of the principal advantages photogrammetry displays over alternative 3-D coordinate measuring devices, such as laser trackers and articulating arms, is that rather than recording data sequentially, point-by-point, the images from which the 3-D coordinates of points are determined capture the entire point field in an instant. This means that both the shape and changes of shape of the object being measured can be determined in near real time. In fact, the position and orientation of several parts can be simultaneously determined at a high data rate to very high accuracy. Moreover, photogrammetric measurement configurations comprising two or more synchronized cameras can perform high accuracy 3-D part measurement in situations where the part is moving or the camera supports are subject to movement or vibration; only the shape of a subset of reference points needs to remain stable. Thus, photogrammetry with synchronized image recording offers the unique capability of operating from moving platforms or in unstable environments, for example when the camera technician needs to record images from high above an unstable factory floor or above a moving automobile assembly line.

A further feature illustrating the versatility of photogrammetry is that, since object shape is being determined in the first instance, there is considerable flexibility in assigning reference coordinate systems. The assignment of both a desired XYZ coordinate system and true scale to the determined 3-D array of measurement points is generally achieved via a small number of reference points, along with known point-to-point distances on scalebars. This is distinct from laser trackers and articulating arms where the initial XYZ coordinate system moves as the instrument moves between measurement stations. From a practical point of view, this means that photogrammetry is generally an ideal 3-D assembly tool in situations where it is required to:

1. dimensionally monitor in near real time an object that is shape invariant, but moving within an unstable environment
2. continuously measure an unstable object or objects within a stable external reference frame.

 

In order to illustrate the applicability of photogrammetry in such cases, four example applications of near real-time dimensional monitoring are now considered. These range from design concepts through to implemented photogrammetric measurement solutions, and all focus upon high accuracy 3-D measurement in dynamic 3-D assembly environments.

Tool tracking

Continual improvement in monitoring the proper usage of manually operated tools is a goal of all manufacturers of large-scale assemblies. One avenue for improvement is through use of an on-line 3-D measurement system that could continuously track the spatial position and orientation of a tool during its application. Consider the case of a drill, for example.  The technology of real-time photogrammetry could potentially be employed to provide continuous positional monitoring, as illustrated in Figure 2. As the drill is tracked in real time, a feedback system would inform the operator as to whether it is in the correct position, with the correct orientation. The continuous monitoring would also allow an audit trail for the work session.

By the standards of large scale vision metrology, the accuracy tolerances in such applications are relatively modest, though there are challenges of a more practical nature to be overcome. For example, the monitoring points on the tool would need to be always “seen” from two or more camera stations, so the camera configuration would need careful design to ensure maintenance of lines of sight to the tool being tracked. The use of relatively inexpensive on-line industrial cameras will allow multisensor arrangements whereby the tools of interest would be continually visible. The camera positions need not be stable, since their instantaneous position and orientation, along with that of the tool, could be determined by the use of visible reference control points. However, maintenance of platform stability would ensure that no additional control point information would be required.

Figure 2: Illustration of real-time positional monitoring of tool operation via photogrammetry.

100% inspection 100% of the time

Automakers have long needed access to more comprehensive 3-D measurement data on the components of vehicles they assemble. The problem has traditionally been, however, that dimensional inspection should not impede the rate of production, and thus the majority of measurements to support 3-D assembly are conducted off line using only samples of component parts. Much of this offline inspection requires a good deal of human interaction, the use of laboratory-based coordinate measuring machines (CMMs), for example. An ideal alternative 3-D measurement approach would be to measure all components on the assembly line, i.e., 100 percent inspection 100 percent of the time.

Such a dimensional measurement goal is yet to be realized, partly as a consequence of the varying nature of part features requiring inspection. These include tooling points; holes; slots; surface profiles; and gaps and edges, both sharp and rounded. Nevertheless, potential technology solutions exist for the online 3-D measurement of some of these feature classes, with photogrammetry again showing promise as a viable, near real-time automated measurement tool that can deliver partial solutions for this general problem even with today’s technology.

Traditional measurement systems available to automakers include CMMs that can measure some of the features on some of the parts. The ideal online inspection system would measure required features on all of the parts being assembled. A photogrammetry system comprising several digital cameras, an illumination system, a projection system, and specialized object feature extraction software could potentially be installed on the assembly line. Such a system would measure some, maybe even most, of the desired features types on every part. The projection system would project dots and lines onto the component surfaces, and the special illumination system would assist in the detection and measurement of surface points, edges, holes, etc. in the synchronized multiple images covering the measurement volume on the assembly line. As can be inferred, the development of a complete solution to this measurement task has yet to be achieved, but photogrammetric systems offering partial solutions are possible with today’s technology.

Robot guidance and watch-dogging

Nowadays, the cutting, shaping, and finishing of large and complex structures is commonly performed with very large, multi-axis computer numerical controlled (CNC) machines, many of which have integrated lasers to control end-effecter positioning. Over time, positioning sensors can suffer from drift, which even when small in magnitude, can be significant in high accuracy machining tasks. Rectification of departures from design positioning is invariably left to the machine operator who needs to take corrective action before an out-of-tolerance condition or even part damage occurs.

Real-time positioning of both the part and the machine head is one means to avoid such problems. One option for online positional monitoring is photogrammetry. Two or more cameras can be configured to track both the position of the end effecter of the CNC machine within its own coordinate system, and more important, within the 3-D coordinate system of the part being machined. Targets need to be positioned on the end effecter to track within all its axes. However, if recently developed photogrammetric processing tools are adopted, all six degrees of freedom (three translations and three rotations) can be determined in real time within a target array where subsets of points are seen from only one camera position. This can be visualized in Figure 2 if one imagines that subsets of the targets on the drills, which have known relative positions, would be recorded in only a single image, though there may be several cameras involved.  

As with the case of the tool tracking, stable reference points on, say, the floor or machine tool supports are required if the camera stations are unstable, which they are permitted to be, whereas no control points are needed if the camera platforms are stable. In some complex machining cases, the actual part that is being machined needs to move during the process so that the CNC machine can access all areas of the part, as exemplified by the rotation of large cylindrical objects. In this case, the vision-based monitoring system has quite a complicated task to perform. Not only must camera positions be determined relative to the real world (the reference system of the whole machining assembly), but the part coordinates must be transformed into the same reference system in real-time, as the part moves.

In these instances, there are effectively two objects whose 3-D coordinates need to be tracked, the part being worked and the end effecter of the CNC machine. Moreover, the tracking needs to be accomplished from the photogrammetric measurement system, which is also potentially subject to movement, or to vibration at the very least. The use of highly-synchronized cameras and control targets can compensate for such platform instability and thus achieve high-accuracy, real-time dimensional monitoring of the machining operation. If the relative position of the CNC machine to the part falls out of tolerance, a feedback loop allows for online corrective action in the CNC positioning.

Guidance of large assembly joins

Recent developments in aircraft manufacturing have brought about a requirement to dynamically control, with a great deal of accuracy, the joining process for very large assemblies such as fuselage sections. As indicated by Figure 3, the dimensional monitoring of such operations can be performed via online, multi-camera photogrammetry systems. The coordinates of targeted arrays of points on both abutting sections can be determined at desired time intervals so as to control, in near real time, the incremental positional movements undertaken in the joining process. Once again, stability of the cameras forming the photogrammetric network is not required, so long as stable reference points are imaged. These may well be built into the factory floor, for example. This photogrammetry solution can also be employed for related dynamic 3-D assembly tasks such as joining the main wings and empennage (the tail assembly) to the fuselage. An added benefit to such a system is its ability to perform symmetry checks at the same time.

Figure 3: Dynamic alignment of fuselage sections via real-time photogrammetry.

Photogrammetry–an optimal 3-D assembly solution

As has been illustrated in the briefly summarized 3-D assembly applications, photogrammetry displays a number of notable attributes that make it ideal for on-line 3-D measurement tasks in large scale manufacturing, where high accuracy coordinate determination is required in near real time in order that the measured data can be used for dynamic process control. As well as being a very flexible measurement technology that can accommodate near instantaneous measurement of either dense target arrays or features, or even just a modest number of critical points, photogrammetry can operate in unstable environments which are the norm within large-scale manufacturing and assembly. This feature renders photogrammetry suitable for complex, often dynamic, 3-D assembly tasks across a broad range of industries in situations where alternative 3-D measurement tasks are invariably unsuitable.

About The Author