This thesis was devoted to the study of the novel methods of communication and encryption using chaotic system in order to improve the existing communication schemes. Some new theoretical properties of chaotic system synchronization was developed, as these methods depend crucially on chaos synchronization phenomena. In particular, new theoretical properties of the so-called generalized Lorenz system has been described. These properties was used to design and systematically analyze the new communication and encryption scheme, called the anti-synchronization chaos shift keying (ACSK) implemented via the generalized Lorenz system. Further, analysis of dynamical properties of generalized Lorenz system enabled study of its synchronization within dynamical complex networks for possible communication. More specifically, different chaotic communication techniques that can be implemented with and without synchronization have been studied in the present thesis. Encryption methods based on the properties of chaos are reviewed. The main contribution of the thesis is the novel modulation scheme called the anti-synchronization chaos shift keying. ACSK digital communication method has potential of introducing a high degree of security at a low receiver complexity. At the same time, it requires reasonable amount of data to encrypt a single bit, thereby making revolutionary possibility of practical and realistic use of continuous time chaotic system for digital data encryption. The thesis implements the ACSK scheme by using the so-called generalized Lorenz system (GLS) family. GLS has been introduced and studied relatively recently, [20; 81; 10], nevertheless, its using to ACSK implementation, and further theoretical analysis was performed in this thesis. The ideas about the communication using generalized Lorenz system via their synchronization are generalized to study the synchronization of complex networks of chaotic systems. Namely, interesting theoretical proof of the exponential synchronization of two generalized Lorenz systems with bi-directional connection has been presented and more complicated networks structure studied numerically. Basic observation here is that the increasing complexity of connections can destabilize the network, stability is maintained by high synchronizing gains and locally only, with decreasing size of stability region.