Previous work in the research field of video surveillance intensively focused on separated aspects of object detection, data association, pattern recognition and system design. In contrast, we propose a holistic approach for object tracking in a self-organizing and distributed smart camera network. Each observation task is represented by a software-agent which improves the tracking performance by collaborative behavior. An object tracking agent detects persons in a video stream and associates them with a trajectory. The pattern recognition agent analyses these trajectories by detecting points of interest within the observation field. These are characterized by a non-deterministic behavior of the moving person. The trajectory points (enriched by the results of the pattern recognition agent) will be used by a configuration agent to align the cameras field of view. We show that this collaboration improves the performance of the observation system by increasing the amount of detected trajectory points by 22%.