Authors,Author(s) ID,Title,Year,Source title,Volume,Issue,Art. No.,Page start,Page end,Page count,Cited by,DOI,Link,Affiliations,Authors with affiliations,Abstract,Author Keywords,Index Keywords,Molecular Sequence Numbers,Chemicals/CAS,Tradenames,Manufacturers,Funding Details,Funding Text 1,Funding Text 2,References,Correspondence Address,Editors,Sponsors,Publisher,Conference name,Conference date,Conference location,Conference code,ISSN,ISBN,CODEN,PubMed ID,Language of Original Document,Abbreviated Source Title,Document Type,Publication Stage,Open Access,Source,EID
"Bhatia H., Paul W., Alajaji F., Gharesifard B., Burlina P.","57204008203;57218937334;7004732407;12646598900;6603713214;","Least kth-order and rényi generative adversarial networks",2021,"Neural Computation","33","9",,"2473","2510",,,"10.1162/neco_a_01416","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85113315947&doi=10.1162%2fneco_a_01416&partnerID=40&md5=b8a79c145d603ab1602a55fdcc54a65c","Department of Mathematics and Statistics, Queens UniversityON  K7L 3N6, Canada; Johns Hopkins University Applied Physics Laboratory, Laurel, MD  20723, United States; Department of Computer Science, Johns Hopkins University, Baltimore, MD  21218, United States","Bhatia, H., Department of Mathematics and Statistics, Queens UniversityON  K7L 3N6, Canada; Paul, W., Johns Hopkins University Applied Physics Laboratory, Laurel, MD  20723, United States; Alajaji, F., Department of Mathematics and Statistics, Queens UniversityON  K7L 3N6, Canada; Gharesifard, B., Department of Mathematics and Statistics, Queens UniversityON  K7L 3N6, Canada; Burlina, P., Johns Hopkins University Applied Physics Laboratory, Laurel, MD  20723, United States, Department of Computer Science, Johns Hopkins University, Baltimore, MD  21218, United States","We investigate the use of parameterized families of information-theoretic measures to generalize the loss functions of generative adversarial networks (GANs) with the objective of improving performance. A new generator loss function, least kth-order GAN (LkGAN), is introduced, generalizing the least squares GANs (LSGANs) by using a kth-order absolute error distortion measure with k ≥ 1 (which recovers the LSGAN loss function when k = 2). It is shown that minimizing this generalized loss function under an (unconstrained) optimal discriminator is equivalent to minimizing the kth-order Pearson-Vajda divergence. Another novel GAN generator loss function is next proposed in terms of Rényi cross-entropy functionals with order α > 0, α ≠ 1. It is demonstrated that this Rényi-centric generalized loss function, which provably reduces to the original GAN loss function as α → 1, preserves the equilibrium point satisfied by the original GAN based on the Jensen-Rényi divergence, a natural extension of the Jensen-Shannon divergence. Experimental results indicate that the proposed loss functions, applied to the MNIST and CelebA data sets, under both DCGAN and StyleGAN architectures, confer performance benefits by virtue of the extra degrees of freedom provided by the parameters k and α, respectively. More specifically, experiments show improvements with regard to the quality of the generated images as measured by the Fréchet inception distance score and training stability. While it was applied to GANs in this study, the proposed approach is generic and can be used in other applications of information theory to deep learning, for example, the issues of fairness or privacy in artificial intelligence. © 2021 Massachusetts Institute of Technology.",,"Deep learning; Degrees of freedom (mechanics); Image enhancement; Adversarial networks; Distortion measures; Equilibrium point; Improving performance; Information theoretic measure; Jensen-Shannon divergence; Natural extension; Performance benefits; Information theory; article; artificial intelligence; deep learning; degree of freedom; entropy; information science; least square analysis; loss of function mutation; privacy; theoretical study",,,,,,,,"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Zheng, X., (2015) TensorFlow: Large-scale machine learning on heterogeneous systems, , https://www.tensorflow.org/; Achille, A., Soatto, S., (2019) Where is the information in a deep neural network?, , arXiv:1905.12213; Alajaji, F., Chen, P.-N., Rached, Z., Csiszár’s cutoff rates for the general hypothesis testing problem (2004) IEEE Transactions on Information Theory, 50 (4), pp. 663-678; Alemi, A. A., Fischer, I., Dillon, J. V., Murphy, K., Deep variational information bottleneck (2017) Proceedings of the 5th International Conference on Learning Representations, pp. 1-19. , https://arxiv.org/pdf/1612.00410.pdf; Arikan, E., An inequality on guessing and its applications to sequential decoding (1996) IEEE Transactions on Information Theory, 42 (1), pp. 99-105; Arjovsky, M., Chintala, S., Bottou, L., Wasserstein generative adversarial networks (2017) Proceedings of the 34th International Conference on Machine Learning, 70, pp. 214-223; Ben-Bassat, M., Raviv, J., Rényi’s entropy and the probability of error (2006) IEEE Transactions on Information Theory, 24 (3), pp. 324-331; Bhatia, H., Paul, W., Alajaji, F., Gharesifard, B., Burlina, P., (2020) Rényi generative adversarial networks, , arXiv:2006.02479; Burlina, P., Joshi, N., Paul, W., Pacheco, K. D., Bressler, N. M., (2020) Addressing artificial intelligence bias in retinal disease diagnostics, , arXiv:2004.13515; Campbell, L. L., A coding theorem and Rényi’s entropy (1965) Information and Control, 9, pp. 423-429; Chen, L., Dai, S., Pu, Y., Zhou, E., Li, C., Su, Q., Carin, L., Symmetric variational autoencoder and connections to adversarial learning (2018) Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, 84, pp. 661-669. , http://proceedings.mlr.press/v84/chen18b.html, PMLR; Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P., InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets (2016) Advances in neural information processing systems, 29, pp. 2172-2180. , D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, & R. Garnett (Eds), –). Red Hook, NY: Curran; Courtade, T. A., Verdú, S., Cumulant generating function of codeword lengths in optimal lossless compression (2014) Proceedings of the IEEE International Symposium on Information Theory, pp. 2494-2498. , Piscataway, NJ: IEEE; Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., Bharath, A. A., Generative adversarial networks: An overview (2018) IEEE Signal Processing Magazine, 35 (1), pp. 53-65; Csiszár, I., Information-type measures of difference of probability distributions and indirect observations (1967) Studia Sci. Math. Hungarica, 2, pp. 299-318; Csiszár, I., Generalized cutoff rates and Rényi’s information measures (1995) IEEE Transactions on Information Theory, 41 (1), pp. 26-34; Engel, E., Dreizler, R. M., (2011) Density functional theory, , Berlin: Springer; Esposito, A. R., Gastpar, M., Issa, I., Robust generalization via α-mutual information (2020) Proceedings of the International Zurich Seminar on Information and Communication, pp. 96-100. , https://doi.org/10.3929/ethz-b-000403224; Farnia, F., Tse, D., A convex duality framework for GANs (2018) Advances in neural information processing systems, 31, pp. 5248-5258. , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds), –). Red Hook, NY: Curran; Goodfellow, I., (2016) NIPS 2016 tutorial: Generative adversarial networks, , arXiv:1701.00160; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Bengio, Y., Generative adversarial nets (2014) Advances in neural information processing systems, 27, pp. 2672-2680. , Ghahramani, M. Welling, C. Cortes, N. Lawrence, & K. Q. Weinberger (Eds), –). Red Hook, NY: Curran; Grover, A., Dhar, M., Ermon, S., Flow-GAN: Combining maximum likelihood and adversarial learning in generative models (2018) Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pp. 3069-3076. , Palo Alto, CA: AAAI; Hamza, A. B., Krim, H., Jensen-Rényi divergence measure: Theoretical and computational perspectives (2003) Proceedings of the IEEE International Symposium on Information Theory, p. 257. , Piscataway, NJ: IEEE; He, Y., Hamza, A. B., Krim, H., A generalized divergence measure for robust image registration (2003) IEEE Transactions on Signal Processing, 51, pp. 1211-1220; Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S., GANstrained by a two time-scale update rule converge to a local Nash equilibrium (2017) Advances in neural information processing systems, 30, pp. 6626-6637. , Guyon, Y. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds)). Red Hook, NY: Curran; Huang, C., Kairouz, P., Chen, X., Sankar, L., Rajagopal, R., (2018) Generative adversarial privacy, , arXiv:1807.05306; Karras, T., Laine, S., Aila, T., A style-based generator architecture for generative adversarial networks (2019) Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410. , Piscataway, NJ: IEEE; Kingma, D. P., Dhariwal, P., Glow: Generative flow with invertible 1 × 1 convolutions (2018) Advances in neural information processing systems, 31, pp. 10215-10224. , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds), –). Red Hook, NY: Curran; Kingma, D. P., Welling, M., Auto-encoding variational Bayes (2014) Proceedings of the 2nd International Conference on Learning Representations, pp. 1-14; Kluza, P. A., On Jensen-Rényi and Jeffreys-Rényi type f-divergences induced by convex functions (2019) Physica A: Statistical Mechanics and Its Applications, 548, pp. 1-10. , https://doi.org/10.1016/j.physa.2019.122527; LeCun, Y., Cortes, C., (1998) MNIST handwritten digit database, , http://yann.lecun.com/exdb/mnist/; Lee, K. S., Tran, N.-T., Cheung, N.-M., (2021) Proceedings of the 2021 IEEE Winter Conference on Applications of Computer Vision, pp. 1-6. , Piscataway, NJ: IEEE; Li, C., Bai, K., Li, J., Wang, G., Chen, C., Carin, L., Adversarial learning of a sampler based on an unnormalized distribution (2019) Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics 2019, 89, pp. 3302-3311. , http://proceedings.mlr.press/v89/li19h.html; Li, Y., Gal, Y., Dropout inference in Bayesian neural networks with alpha-divergences (2017) Proceedings of the 34th International Conference on Machine Learning, 70, pp. 2052-2061; Li, Y., Turner, R. E., Rényi divergence variational inference (2016) Advances in neural information processing systems, 29, pp. 1073-1081. , D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, & R. Garnett (Eds), –). Red Hook, NY: Curran; Liu, Z., Luo, P., Wang, X., Tang, X., Deep learning face attributes in the wild (2015) Proceedings of International Conference on Computer Vision, pp. 1-11. , Piscataway, NJ: IEEE; Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., Smolley, S. P., Least squares generative adversarial networks (2017) Proceedings of the IEEE International Conference on Computer Vision, pp. 1-16. , Piscataway, NJ: IEEE; Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., Smolley, S. P., (2017) On the effectiveness of least squares generative adversarial networks, , arXiv:1712.06391; Mescheder, L., Geiger, A., Nowozin, S., Which training methods for GANs do actually converge? (2018) Proceedings of the 35th International Conference on Machine Learning, 80, pp. 3481-3490; Mwebaze, E., Schneider, P., Schleif, F.-M., Haase, S., Villmann, T., Biehl, M., Divergence based learning vector quantization (2010) Proceedings of the 18th European Symposium on Artificial Neural Networks, pp. 247-252; Nielsen, F., (2019) On a generalization of the Jensen-Shannon divergence, , arXiv:1912.00610; Nielsen, F., Nock, R., On the chi square and higher-order chi distances for approximating f-divergences (2013) IEEE Signal Processing Letters, pp. 10-13; Nowozin, S., Cseke, B., Tomioka, R., f-GAN: Training generative neural samplers using variational divergence minimization (2016) Advances in neural information processing systems, 29, pp. 271-279. , D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, & R. Garnett (Eds), –). Red Hook, NY: Curran; Oord, A. v. d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kavukcuoglu, K., (2016) Wavenet: A generative model for raw audio, , arXiv:1609.03499; Pantazis, Y., Paul, D., Fasoulakis, M., Stylianou, Y., Katsoulakis, M., (2020) Cumulant GAN, , arXiv:2006.06625; Paul, W., Wang, I., Alajaji, F., Burlina, P., Unsupervised discovery, control, and disentanglement of semantic attributes with applications to anomaly detection (2021) Neural Computation, 33 (3), pp. 802-826; Principe, J. C., (2010) Information theoretic learning: Rényi’s entropy and kernel perspectives, , Berlin: Springer; Rached, Z., Alajaji, F., Campbell, L. L., Rényi entropy rate for discrete Markov sources (1999) Proceedings of the 33rd Conference on Information Sciences and Systems, pp. 613-618; Radford, A., Metz, L., Chintala, S., Unsupervised representation learning with deep convolutional generative adversarial networks (2017) Proceedings of the 9th International Conference on Image and Graphics, pp. 97-108; Rényi, A., On measures of entropy and information (1961) Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, 1, pp. 547-561. , Berkeley: University of California, Berkeley; Sarraf, A., Nie, Y., RGAN: Rényi generative adversarial network (2021) SN Computer Science, 2 (1), p. 17; Sason, I., On f-divergences: Integral representations, local behavior, and inequalities (2018) Entropy, 20, pp. 1-32; Tishby, N., Zaslavsky, N., Deep learning and the information bottleneck principle (2015) Proceedings of the 2015 IEEE Information Theory Workshop, pp. 1-5. , Piscataway, NJ: IEEE; Valverde-Albacete, F. J., Peláez-Moreno, C., The case for shifting the Rényi entropy (2019) Entropy, 21, pp. 1-21. , https://doi.org/10.3390/e21010046; van Erwen, T., Harremos, P., Rényi divergence and Kullback-Leibler divergence (2014) IEEE Transactions on Information Theory, 60 (7), pp. 3797-3820; Verdú, S., α-mutual information (2015) Proceedings of the 2015 IEEE Information Theory and Applications Workshop, pp. 1-6. , Piscataway, NJ: IEEE; Wang, Z., She, Q., Ward, T. E., (2020) Generative adversarial networks in computer vision: A survey and taxonomy, , arXiv:1906.01529v3; Wiatrak, M., Albrecht, S. V., Nystrom, A., (2020) Stabilizing generative adversarial network training: A survey, , arXiv:1910.00927v2; Wickstrom, K., Lokse, S., Kampffmeyer, M., Yu, S., Principe, J., Jenssen, R., (2019) Information plane analysis of deep neural networks via matrix-based Rényi’s entropy and tensor kernels, , arXiv:1909.11396; Zaidi, A., Estella-Aguerri, I., Shamai, S., On the information bottleneck problems: Models, connections, applications and information theoretic views (2020) Entropy, 22, pp. 1-36; Zhao, M., Cong, Y., Dai, S., Carin, L., Bridging maximum likelihood and adversarial learning via alpha-divergence (2020) Proceedings of the 34th AAAI Conference on Artificial Intelligence, 34, pp. 6901-6908. , Palo Alto, CA: AAAI",,,,"MIT Press Journals",,,,,08997667,,,"34412112","English","Neural Comp.",Letter,"Final","All Open Access, Bronze, Green",Scopus,2-s2.0-85113315947
"Na B., Son S.","56594540400;55835324700;","Prediction of atmospheric motion vectors around typhoons using generative adversarial network",2021,"Journal of Wind Engineering and Industrial Aerodynamics","214",,"104643","","",,1,"10.1016/j.jweia.2021.104643","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85105593313&doi=10.1016%2fj.jweia.2021.104643&partnerID=40&md5=0877f91cc474ac4fe7c6eefdd5b9799e","Future and Fusion Lab for Architectural, Civil and Environmental Engineering, Korea University, Seoul, 02841, South Korea; School of Civil, Environmental and Architectural Engineering, Korea University, Seoul, 02841, South Korea","Na, B., Future and Fusion Lab for Architectural, Civil and Environmental Engineering, Korea University, Seoul, 02841, South Korea; Son, S., School of Civil, Environmental and Architectural Engineering, Korea University, Seoul, 02841, South Korea","In this study, atmospheric motion vectors (AMVs) were derived from the satellite images predicted using a generative adversarial network (GAN) and a deep multi-scale frame prediction algorithm. The GAN was trained and tested with a sequence of the satellite images of a COMS satellite infrared-window channel under the 68 tropical cyclones. The inputs of the consecutive satellite images with 15-min interval were then processed using the trained GAN model to generate satellite images in the next time steps. To further enhance the model's predictability, particle image velocimetry based on the theory of cross-correlation schemes was employed to the GAN-generated satellite image sequence and AMVs were produced. The GAN-derived AMVs were validated with the wind fields based on the numerical weather prediction (NWP) and radiosonde observations. The comparisons showed that the GAN-derived AMVs depicted the structure of atmospheric circulations with a certain level of accuracy. Through comparison with the radiosonde observations, the root-mean-square error and the wind speed bias of the GAN-derived AMVs were comparable to, and even smaller than those of the NWP-derived wind fields. The current approach may enhance the accuracy in predicting short-term wind velocity fields, which in turn may provide more realistic inputs in storm surge modeling. © 2021","Atmospheric motion vectors; Generative adversarial network; Particle image velocimetry; Satellite images; Wind velocity","Flow visualization; Hurricanes; Image enhancement; Mean square error; Radiosondes; Satellites; Storms; Velocimeters; Velocity; Weather forecasting; Wind; Adversarial networks; Atmospheric motion vectors; Generative adversarial network; Image velocimetry; Numerical weather prediction; Particle images; Radiosonde observations; Satellite images; Wind field; Wind velocities; Velocity measurement",,,,,"National Research Foundation of Korea, NRF: 2020R1C1C100513311, NRF-2019H1D3A1A0107072 2","This research was supported by the National Research Foundation of Korea ( NRF-2019H1D3A1A0107072 2) and the National Research Foundation of Korea ( 2020R1C1C100513311 ).",,"Abidi, M.A., Gonzalez, R.C., Cloud motion measurement from radar image sequences (1988) Proc. SPIE, 846, pp. 54-60; Alemany, S., Beltran, J., Perez, A., Ganzfried, S., Predicting Hurricane Trajectories Using a Recurrent Neural Network (2018), arXiv:1802.02548vol. 2; Bedka, K.M., Mecikalski, J.R., Application of satellite-derived atmospheric motion vectors for estimating mesoscale flows (2005) J. Appl. Meteorol., 44, pp. 1761-1772; Berger, H., Langland, R., Velden, C.S., Reynolds, C.A., Pauley, P.M., Impact of enhanced satellite-derived atmospheric motion vector observations on numerical tropical track forecasts in the western North Pacific during TPARC/TCS-08 (2011) J. Appl. Meteor. Climatol., 50, pp. 2309-2318; Borde, R., Doutriaux-Boucher, M., Dew, G., Carranza, M., A direct link between feature tracking and height assignment of operational EUMETSAT atmospheric motion vectors (2014) J. Atmos. Ocean. Technol., 31, pp. 33-46; Bresky, W.C., Daniels, J.M., Bailey, A.A., Wanzong, S.T., New methods toward minimizing the slow speed bias associated with atmospheric motion vectors (2012) J. Appl. Meteor. Climatol., 51, pp. 2137-2151; Cardone, V.J., Cox, A.T., Tropical cyclone wind field forcing for surge models: critical issues and sensitivities (2009) Nat. Hazards, 51 (1), pp. 29-47; Cherubini, T., Businger, S., Velden, C., Ogasawara, R., The impact of satellite-derived atmospheric motion vectors on mesoscale forecasts over Hawaii (2006) Mon. Weather Rev., 134, pp. 2009-2020; Chu, D., Zhang, J., Wu, Y., Jiao, X., Qian, S., Sensitivities of modelling storm surge to bottom friction, wind drag coefficient, and meteorological product in the East China Sea (2019) Estuar. Coast Shelf Sci., 231, p. 106460; Chuang, W., Chou, C., Chang, K., Chang, Y., Chin, H., Atmospheric motion vectors derived from an infrared window channel of a geostationary satellite using particle image velocimetry (2019) J. Appl. Meteor. Climatol., 58, pp. 199-211; Endlich, R.M., Wolf, D.E., Automatic cloud tracking applied to GOES and METEOSAT observations (1981) J. Appl. Meteorol., 20, pp. 309-319; Endlich, R.M., Wolf, D.E., Hall, D.J., Brain, A.E., Use of a pattern recognition technique for determining cloud motions from sequences of satellite photographs (1971) J. Appl. Meteorol., 10, pp. 105-117. , 2.0.CO;2; Fleming, J.G., Fulcher, C.W., Luettich, R.A., Estrade, B.D., Allen, G.D., Winer, H.S., A real time storm surge forecasting system using ADCIRC (2008) Proc., 10th Int. Cong. On Estuarine and Coastal Modeling, pp. 893-912. , Reston, VA: ASCE Goodfellow, I. et al. Generative adversarial networks. arXiv:1406.2661v1 (2014); Giffard-Roisin, S., Yang, M., Charpiat, G., Kégl, B., Monteleoni, C., Fused deep learning for hurricane track forecast from reanalysis data (2018) Proc. Clim. Inform. Workshop; Giffard-Roisin, S., Yang, M., Charpiat, G., Kégl, B., Monteleoni, C., Deep learning for hurricane track forecasting from aligned spatio-temporal climate datasets (2018) Proc. 32nd Conf. NeurIPS.; Github, Adversarial video generation (2019), https://github.com/dyelax/Adversarial_Video_Generation; Goerss, J.S., Impact of satellite observations on the tropical cyclone track forecasts of the navy operational global atmospheric prediction system (2009) Mon. Weather Rev., 137, pp. 41-50; Harper, B., Kepert, J., Ginger, J., World meteorological organization guidelines for converting between various wind averaging periods in tropical cyclone conditions (2008), https://library.wmo.int/doc_num.php?explnum_id=290, October 2008, Available from:; He, Y.C., Chan, P.W., Li, Q.S., Observations of vertical wind profiles of tropical cyclones at coastal areas (2016) J. Wind Eng. Ind. Aerod., 152, pp. 1-14; He, J.Y., Li, Q.S., Chan, P.W., Reduced gust factor for extreme tropical cyclone winds over ocean (2021) J. Wind Eng. Ind. Aerod., 208, p. 104445; Holland, G.J., An analytic model of the wind and pressure profiles in hurricanes (1980) Mon. Weather Rev., 108 (8), pp. 1212-1218; Holmlund, K., The utilization of statistical properties of satellite-derived atmospheric motion vectors to derive quality indicators (1998) Weather Forecast., 13, pp. 1093-1104; Hong, S., Kim, S., Joh, M., Song, A., Globenet: Convolutional Neural Networks for Typhoon Eye Tracking from Remote Sensing Imagery (2017), arXiv:1708.03417vol. 1; Houston, S.H., Shaffer, W.A., Powell, M.D., Chen, J., Comparisons of HRD and SLOSH surface wind fields in hurricanes: implications for storm surge modeling (1999) Weather Forecast., 14 (5), pp. 671-686; Hu, G., Liu, L., Tao, D., Song, J., Tse, K.T., Kwok, K.C.S., Deep learning-based investigation of wind pressures on tall building under interference effects (2020) J. Wind Eng. Ind. Aerod., 201, p. 104138; Hwang, S., Son, S., Lee, C., Yoon, H.D., Quantitative assessment of inundation risks from physical contributors associated with future storm surges: a case study of Typhoon Maemi (2003) (2020) Nat. Hazards, 104 (2), pp. 1389-1411; Kim, Y., Son, S., Jung, T., Gallien, T., An analytical and numerical study of a vertically discretized multi-paddle wavemaker for generating free surface and internal waves (2021) Coast Eng., p. 103840; Kordmahalleh, M.M., Sefidmazgi, M.G., Homaifar, A., A sparse recurrent neural network for trajectory prediction of atlantic hurricanes (2016) Proc. Genet. Evol. Comput. Conf., pp. 957-964; Kossin, J.P., A global slowdown of tropical-cyclone translation speed (2018) Nature, 558 (7708), pp. 104-107; Langland, R.H., Velden, C., Pauley, P.M., Berger, H., Impact of satellite-derived rapid-scan wind observations on numerical model forecasts of Hurricane Katrina (2009) Mon. Weather Rev., 137, pp. 1615-1622; Le Marshall, J., Coauthors, Himawari-8 atmospheric motion vectors—operational generation and assimilation (2017) J. South. Hemisphere Earth Syst. Sci., 67, pp. 12-24; Lee, R.S.T., Liu, J.N.K., Tropical cyclone identification and tracking system using integrated neural oscillatory elastic graph matching and hybrid RBF network track mining techniques (2000) IEEE Trans. Neural Network., 11, pp. 680-689; Lee, C., Hwang, S., Do, K., Son, S., Increasing flood risk due to river runoff in the estuarine area during a storm landfall (2019) Estuar. Coast Shelf Sci., 221, pp. 104-118; Lim, H.J., Chang, K.A., Huang, Z.C., Na, B., Experimental study on plunging breaking waves in deep water (2015) J. Geophys. Res. Oceans, 120, pp. 2007-2049; Lotter, W., Kreiman, G., Cox, D., Deep predictive coding networks for video prediction and unsupervised learning (2017) in Proc. Int. Conf. Learn. Rep., pp. 1-18. , arXiv:1605.08104; Mathieu, M., Couprie, C., LeCun, Y., Deep Multi-Scale Video Prediction beyond Mean Square Error (2015), arXiv: 1511; Mecikalski, J.R., MacKenzie, W.M., König, M., Muller, S., Cloud-top properties of growing cumulus prior to convective initiation as measured by meteosat second generation. Part II: use of visible reflectance (2010) J. Appl. Meteor. Climatol., 49, pp. 2544-2558; Menzel, W.P., Cloud tracking with satellite imagery: from the pioneering work of Ted Fujita to the present (2001) Bull. Am. Meteorol. Soc., 82, pp. 33-48; Na, B., Chang, K.-A., Huang, Z.-C., Lim, H.-J., Turbulent flow field and air entrainment in laboratory plunging breaking waves (2016) J. Geophys. Res. Oceans, 121, pp. 2980-3009; Nieman, S.J., Menzel, W.P., Hayden, C.M., Gray, D., Wanzong, S.T., Velden, C.S., Daniels, J., Fully automated cloud-drift winds in NESDIS operations (1997) Bull. Am. Meteorol. Soc., 78, pp. 1121-1134; Oh, S.M., Borde, R., Carranza, M., Shin, I.-C., Development and intercomparison study of an atmospheric motion vector retrieval algorithm for GEO-KOMPSAT-2A (2019) Remote Sens., 11, p. 2054. , 2019; Powell, M.D., Murillo, S., Dodge, P., Uhlhorn, E., Gamache, J., Cardone, V., Cox, A., Fleur, R., Reconstruction of Hurricane Katrina's wind fields for storm surge and wave hindcasting (2010) Ocean Eng., 37 (1), pp. 26-36; Rüttgers, M., Lee, S., Jeon, S., Prediction of a typhoon track using a generative adversarial network and satellite images (2019) Sci. Rep., 9, p. 6057; Ryu, Y., Chang, K.A., Lim, H.J., Use of bubble image velocimetry for measurement of plunging wave impinging on structure and associated greenwater (2005) Meas. Sci. Technol., 16, pp. 1945-1953; Schmetz, J., Holmlund, K., Hoffman, J., Strauss, B., Mason, B., Gaertner, V., Koch, A., Van De Berg, L., Operational cloud-motion winds from meteosat infrared images (1993) J. Appl. Meteorol., 32, pp. 1206-1225; Theunissen, R., Scarano, F., Riethmuller, M.L., An adaptive sampling and windowing interrogation method in PIV (2007) Meas. Sci. Technol., 18, pp. 275-287; Thielicke, W., Stamhuis, E.J., PIVlab – towards user-friendly, affordable and accurate digital particle image velocimetry in MATLAB (2014) J. Open Res. Software, 2 (1), p. e30; Torres, M.J., Reza Hashemi, M., Hayward, S., Spaulding, M., Ginis, I., Grilli, S.T., Role of hurricane wind models in accurate simulation of storm surge and waves (2019) J. Waterw. Port, Coast. Ocean Eng., 145 (1); Vega-Riveros, J.F., Jabbour, K., Review of motion analysis techniques (1989) IEE Proc., I, 136, pp. 397-404; Velden, C., Coauthors, Recent innovations in deriving tropospheric winds from meteorological satellites (2005) Bull. Am. Meteorol. Soc., 86, pp. 205-224; Velden, C.S., Hayden, C.M., Nieman, S.J., Menzel, W.P., Wanzong, S., Goerss, J.S., Upper-tropospheric winds derived from geostationary satellite water vapor observations (1997) Bull. Am. Meteorol. Soc., 78, pp. 173-195; Velden, C., Lewis, W.E., Bresky, W., Stettner, D., Daniels, J., Wanzong, S., Assimilation of high-resolution satellite-derived atmospheric motion vectors: impact on HWRF forecasts of tropical cyclone track and intensity (2017) Mon. Weather Rev., 145, pp. 1107-1125; Weckwerth, T.M., Parsons, D.B., A review of convection initiation and motivation for IHOP_2002 (2006) Mon. Weather Rev., 134, pp. 5-22; Wieneke, B., Pfeiffer, K., Adaptive PIV with variable interrogation window size and shape (2010) 15th Int. Symp. On Applications of Laser Techniques to Fluid Mechanics Lisbon, Portugal, , Lisbon Symposia; Wolf, D.E., Hall, D.J., Endlich, R.M., Experiments in automatic cloud tracking using SMS-GOES data (1977) J. Appl. Meteorol., 16, pp. 1219-1230; Wu, T.-C., Velden, C.S., Majumdar, S.J., Liu, H., Anderson, J.L., Understanding the influence of assimilating subsets of enhanced atmospheric motion vectors on numerical analyses and forecasts of tropical cyclone track and intensity with an ensemble Kalman filter (2015) Mon. Weather Rev., 143, pp. 2506-2531; Zhang, Y., Chandra, R., Gao, J., Cyclone track prediction with matrix neural networks (2018) International Joint Conference on Neural Networks","Son, S.; School of Civil, South Korea; email: sson@korea.ac.kr",,,"Elsevier B.V.",,,,,01676105,,,,"English","J. Wind Eng. Ind. Aerodyn.",Article,"Final","",Scopus,2-s2.0-85105593313
"Domínguez-Rodrigo M., Fernández-Jaúregui A., Cifuentes-Alcobendas G., Baquedano E.","6603679662;57224679477;57212305257;36237883200;","Use of generative adversarial networks (Gan) for taphonomic image augmentation and model protocol for the deep learning analysis of bone surface modifications",2021,"Applied Sciences (Switzerland)","11","11","5237","","",,,"10.3390/app11115237","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85108162503&doi=10.3390%2fapp11115237&partnerID=40&md5=7e3c3c13bdbe6a2abc4ae438c1e6b300","Institute of Evolution in Africa (IDEA), Alcalá University, Covarrubias 36, Madrid, 28010, Spain; Area of Prehistory (Department History and Philosophy), University of Alcalá, Alcalá de Henares, 28801, Spain; Regional Archaeological Museum of Madrid, Plaza de las Bernardas s/n, Alcalá de Henares, 28001, Spain","Domínguez-Rodrigo, M., Institute of Evolution in Africa (IDEA), Alcalá University, Covarrubias 36, Madrid, 28010, Spain, Area of Prehistory (Department History and Philosophy), University of Alcalá, Alcalá de Henares, 28801, Spain; Fernández-Jaúregui, A., Institute of Evolution in Africa (IDEA), Alcalá University, Covarrubias 36, Madrid, 28010, Spain; Cifuentes-Alcobendas, G., Institute of Evolution in Africa (IDEA), Alcalá University, Covarrubias 36, Madrid, 28010, Spain, Area of Prehistory (Department History and Philosophy), University of Alcalá, Alcalá de Henares, 28801, Spain; Baquedano, E., Institute of Evolution in Africa (IDEA), Alcalá University, Covarrubias 36, Madrid, 28010, Spain, Regional Archaeological Museum of Madrid, Plaza de las Bernardas s/n, Alcalá de Henares, 28001, Spain","Deep learning models are based on a combination of neural network architectures, optimization parameters and activation functions. All of them provide exponential combinations whose computational fitness is difficult to pinpoint. The intricate resemblance of the microscopic features that are found in bone surface modifications make their differentiation challenging, and determining a baseline combination of optimizers and activation functions for modeling seems necessary for computational economy. Here, we experiment with combinations of the most resolutive activation functions (relu, swish, and mish) and the most efficient optimizers (stochastic gradient descent (SGD) and Adam) for bone surface modification analysis. We show that despite a wide variability of outcomes, a baseline of relu–SGD is advised for raw bone surface modification data. For imbalanced samples, augmented datasets generated through generative adversarial networks are implemented, resulting in balanced accuracy and an inherent bias regarding mark replication. In summary, although baseline procedures are advised, these do not prevent to overcome Wolpert’s “no free lunch” theorem and extend it beyond model architectures. © 2021 by the authors. Licensee MDPI, Basel, Switzerland.","Activation function; Computer vision; Generative adversarial networks; Neural networks; Optimizer; Taphonomy",,,,,,"HAR2017-82463-C4-1-P

Ministerio de Ciencia, Innovación y Universidades, MCIU","Funding: Ministry of Education, Science and Universities, Spain (grant: HAR2017-82463-C4-1-P).","Acknowledgments: We thank the Spanish Ministry of Education, Science and Universities for funding this research (HAR2017-82463-C4-1-P). We also appreciate the constructive comments made by three reviewers. We would like to express our thanks to M. A. Maté-González for having invited us to participate in this Special Issue.","Domínguez-Rodrigo, M., Cifuentes-Alcobendas, G., Jiménez-García, B., Abellán, N., Pizarro-Monzo, M., Organista, E., Baquedano, E., Artificial intelligence provides greater accuracy in the classification of modern and ancient bone surface modifications (2020) Sci. Rep, 10, p. 18862. , [CrossRef] [PubMed]; Cifuentes-Alcobendas, G., Domínguez-Rodrigo, M., Deep learning and taphonomy: High accuracy in the classification of cut marks made on fleshed and defleshed bones using convolutional neural networks (2019) Sci. Rep, 9, p. 18933. , [CrossRef] [PubMed]; Pizarro-Monzo, M., Domínguez-Rodrigo, M., Dynamic modification of cut marks by trampling: Temporal assessment through the use of mixed-effect regressions and deep learning methods (2020) Archaeol. Anthropol. Sci, 12, p. 4. , [CrossRef]; Abellán, N., Jiménez-García, B., Aznarte, J., Baquedano, E., Domínguez-Rodrigo, M., Deep learning classification of tooth scores made by different carnivores: Achieving high accuracy when comparing African carnivore taxa and testing the hominin shift in the balance of power (2021) Archaeol. Anthropol. Sci, 13, p. 31. , [CrossRef]; Jiménez-García, B., Aznarte, J., Abellán, N., Baquedano, E., Domínguez-Rodrigo, M., Deep learning improves taphonomic resolution: High accuracy in differentiating tooth marks made by lions and jaguars (2020) J. R. Soc. Interface, 17, p. 20200446. , [CrossRef]; Jiménez-García, B., Abellán, N., Baquedano, E., Cifuentes-Alcobendas, G., Domínguez-Rodrigo, M., Corrigendum to “Deep learning improves taphonomic resolution: High accuracy in differentiating tooth marks made by lions and jaguars (2020) J. R. Soc. Interface, 17, p. 20200782. , [CrossRef]; Chollet, F., (2017) Deep Learning with Python, p. 361. , Manning Publications Company: New York, NY, USA, ISBN 9781617294433; Shorten, C., Khoshgoftaar, T.M., A survey on Image Data Augmentation for Deep Learning (2019) J. Big Data, 6, p. 60. , [CrossRef]; Mikolajczyk, A., Grochowski, M., Data augmentation for improving deep learning in image classification problem Proceedings of the 2018 International Interdisciplinary PhD Workshop (IIPhDW), pp. 117-122. , Swinoujście, Poland, 9–12 May 2018; Zhang, W., Kinoshita, Y., Kiya, H., Image-Enhancement-Based Data Augmentation for Improving Deep Learning in Image Classification Problem Proceedings of the 2020 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-Taiwan), pp. 1-2. , Taoyuan, Taiwan, 28–30 September 2020; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative Adversarial Nets (2014) Proceedings of the Advances in Neural Information Processing Systems, 27, pp. 2672-2680. , Montreal, QC, Canada, 8–13 December Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K.Q., Eds.; Curran Associates, Inc.: Red Hook, NY, USA, 2014; Langr, J., Bok, V., (2019) GANs in Action: Deep learning with Generative Adversarial Networks, , Manning Publications Company: New York, NY, USA, ISBN 9781617295560; Yi, X., Walia, E., Babyn, P., Generative adversarial network in medical imaging: A review (2019) Med. Image Anal, 58, p. 101552. , [CrossRef]; Sun, Y., Yuan, P., Sun, Y., MM-GAN: 3D MRI Data Augmentation for Medical Image Segmentation via Generative Adversarial Networks Proceedings of the 2020 IEEE International Conference on Knowledge Graph (ICKG), pp. 227-234. , Nanjing, China, 9–11 August 2020; Lan, L., You, L., Zhang, Z., Fan, Z., Zhao, W., Zeng, N., Chen, Y., Zhou, X., Generative Adversarial Networks and Its Applications in Biomedical Informatics (2020) Front Public Health, 8, p. 164. , [CrossRef] [PubMed]; Chang, Q., Qu, H., Zhang, Y., Sabuncu, M., Chen, C., Zhang, T., Metaxas, D.N., Synthetic learning: Learn from distributed asynchronized discriminator gan without sharing medical image data Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13856-13866. , Seattle, DC, USA, 13–19 June 2020; Goodfellow, I., Bengio, Y., Courville, A., (2016) Deep Learning, , MIT Press: Cambridge, MA, USA, ISBN 9780262337373; Domínguez-Rodrigo, M., de Juana, S., Galán, A.B., Rodríguez, M., A new protocol to differentiate trampling marks from butchery cut marks (2009) J. Archaeol. Sci, 36, pp. 2643-2654. , [CrossRef]; Brownlee, J., (2017) Deep Learning with Python: Develop Deep Learning Models on Theano and TensorFlow Using Keras; Machine Learning Mas-tery, , https://books.google.rs/books/about/Deep_Learning_With_Python.html?id=K-ipDwAAQBAJ&printsec=frontcover&source=kp_read_button&redir_esc=y#v=onepage&q&f=false, (accessed on 3 June 2021); Brownlee, J., (2018) Better Deep Learning: Train Faster, Reduce Overfitting, and Make Better Predictions; Machine Learning Mastery, , https://books.google.rs/books/about/Better_Deep_Learning.html?id=T1-nDwAAQBAJ&printsec=frontcover&source=kp_read_button&redir_esc=y#v=onepage&q&f=false, (accessed on 3 June 2021); Eger, S., Youssef, P., Gurevych, I., (2019) Is it Time to Swish? Comparing Deep Learning Activation Functions Across NLP tasks, , arXiv arXiv:1901.02671; Jinsakul, N., Tsai, C.-F., Tsai, C.-E., Wu, P., Enhancement of Deep Learning in Image Classification Performance Using Xception with the Swish Activation Function for Colorectal Polyp Preliminary Screening (2019) Sci. China Ser. A Math, 7, p. 1170. , [CrossRef]; Misra, D., (2019) Mish: A Self Regularized Non-Monotonic Activation Function, , arXiv arXiv:1908.08681; Kingma, D.P., Ba, J., (2014) Adam: A Method for Stochastic Optimization, , arXiv arXiv:1412.6980; Nagarajan, R., Scutari, M., Lèbre, S., (2013) Bayesian Networks in R, 122, pp. 125-127. , Springer: New York, NY, USA; Scutari, M., Denis, J.-B., (2014) Bayesian Networks: With Examples in R, , CRC Press: Boca Raton, FL, USA, ISBN 9781482225587; Hong, Y., Niu, L., Zhang, J., Zhao, W., Fu, C., Zhang, L., F2GAN: Fusing-and-Filling GAN for Few-shot Image Generation (2020) Proceedings of the 28th ACM International Conference on Multimedia, pp. 2535-2543. , Association for Computing Machinery: New York, NY, USA, ISBN 9781450379885; Antoniou, A., Storkey, A., Edwards, H., (2017) Data Augmentation Generative Adversarial Networks, , arXiv arXiv:1711.04340; Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Wang, Z., Photo-realistic single image super-resolution using a generative adversarial network Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681-4690. , Honolulu, HI, USA, 21–26 July 2017; Bourgeon, L., Burke, A., Higham, T., Earliest Human Presence in North America Dated to the Last Glacial Maximum: New Radiocarbon Dates from Bluefish Caves, Canada (2017) PLoS ONE, 12, p. e0169486. , [CrossRef]; Gommery, D., Ramanivosoa, B., Faure, M., Guérin, C., Kerloc’h, P., Sénégas, F., Randrianantenaina, H., Les plus anciennes traces d’activités anthropiques de Madagascar sur des ossements d’hippopotames subfossiles d’Anjohibe (Province de Mahajanga) (2011) Comptes Rendus Palevol, 10, pp. 271-278. , [CrossRef]; Anderson, A., Clark, G., Haberle, S., Higham, T., Nowak-Kemp, M., Prendergast, A., Radimilahy, C., Schwenninger, J.-L., New evidence of megafaunal bone damage indicates late colonization of Madagascar (2018) PLoS ONE, 13, p. e0204368. , [CrossRef] [PubMed]; Hansford, J., Wright, P.C., Rasoamiaramanana, A., Pérez, V.R., Godfrey, L.R., Errickson, D., Thompson, T., Turvey, S.T., Early Holocene human presence in Madagascar evidenced by exploitation of avian megafauna (2018) Sci. Adv, 4, p. eaat6925. , [CrossRef] [PubMed]; Espigares, M.P., Patrocinio Espigares, M., Palmqvist, P., Guerra-Merchán, A., Ros-Montoya, S., García-Aguilar, J.M., Rodríguez-Gómez, G., Martínez-Navarro, B., The earliest cut marks of Europe: A discussion on hominin subsistence patterns in the Orce sites (Baza basin, SE Spain) (2019) Sci. Rep, 9, pp. 1-13. , [CrossRef]; Wolpert, D.H., The Existence of A Priori Distinctions Between Learning Algorithms (1996) Neural Comput, 8, pp. 1391-1420. , [CrossRef]","Domínguez-Rodrigo, M.; Institute of Evolution in Africa (IDEA), Covarrubias 36, Spain; email: manuel.dominguezr@uah.es",,,"MDPI AG",,,,,20763417,,,,"English","Appl. Sci.",Article,"Final","All Open Access, Gold",Scopus,2-s2.0-85108162503
"Shirasaki M., Moriwaki K., Oogi T., Yoshida N., Ikeda S., Nishimichi T.","55330440400;57200962420;55557539900;7403004787;7404042551;16176212100;","Noise reduction for weak lensing mass mapping: An application of generative adversarial networks to Subaru Hyper Suprime-Cam first-year data",2021,"Monthly Notices of the Royal Astronomical Society","504","2",,"1825","1839",,,"10.1093/mnras/stab982","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85107747502&doi=10.1093%2fmnras%2fstab982&partnerID=40&md5=e404aab053b67646641e5eaaa4a89e85","National Astronomical Observatory of Japan, Mitaka, Tokyo, 181-8588, Japan; The Institute of Statistical Mathematics, Tachikawa, Tokyo, 190-8562, Japan; Department of Physics, University of Tokyo, Tokyo, 113-0033, Japan; Institute of Management and Information Technologies, Chiba University, Chiba, 263-8522, Japan; Kavli Institute for the Physics and Mathematics of the Universe (WPI), University of Tokyo, Kashiwa, Chiba, 277-8583, Japan; Institute for Physics of Intelligence, University of Tokyo, Tokyo, 113-0033, Japan; Research Center for the Early Universe, Faculty of Science, University of Tokyo, Tokyo, 113-0033, Japan; Department of Statistical Science, Graduate University for Advanced Studies, 10-3 Midori-cho, Tachikawa, Tokyo, 190-8562, Japan; Center for Gravitational Physics, Yukawa Institute for Theoretical Physics, Kyoto University, Kyoto, 606-8502, Japan","Shirasaki, M., National Astronomical Observatory of Japan, Mitaka, Tokyo, 181-8588, Japan, The Institute of Statistical Mathematics, Tachikawa, Tokyo, 190-8562, Japan; Moriwaki, K., Department of Physics, University of Tokyo, Tokyo, 113-0033, Japan; Oogi, T., Institute of Management and Information Technologies, Chiba University, Chiba, 263-8522, Japan; Yoshida, N., Department of Physics, University of Tokyo, Tokyo, 113-0033, Japan, Kavli Institute for the Physics and Mathematics of the Universe (WPI), University of Tokyo, Kashiwa, Chiba, 277-8583, Japan, Institute for Physics of Intelligence, University of Tokyo, Tokyo, 113-0033, Japan, Research Center for the Early Universe, Faculty of Science, University of Tokyo, Tokyo, 113-0033, Japan; Ikeda, S., The Institute of Statistical Mathematics, Tachikawa, Tokyo, 190-8562, Japan, Department of Statistical Science, Graduate University for Advanced Studies, 10-3 Midori-cho, Tachikawa, Tokyo, 190-8562, Japan; Nishimichi, T., Kavli Institute for the Physics and Mathematics of the Universe (WPI), University of Tokyo, Kashiwa, Chiba, 277-8583, Japan, Center for Gravitational Physics, Yukawa Institute for Theoretical Physics, Kyoto University, Kyoto, 606-8502, Japan","We propose a deep-learning approach based on generative adversarial networks (GANs) to reduce noise in weak lensing mass maps under realistic conditions. We apply image-to-image translation using conditional GANs to the mass map obtained from the first-year data of Subaru Hyper Suprime-Cam (HSC) Survey. We train the conditional GANs by using 25 000 mock HSC catalogues that directly incorporate a variety of observational effects. We study the non-Gaussian information in denoised maps using one-point probability distribution functions (PDFs) and also perform matching analysis for positive peaks and massive clusters. An ensemble learning technique with our GANs is successfully applied to reproduce the PDFs of the lensing convergence. About 60 per cent of the peaks in the denoised maps with height greater than 5σ have counterparts of massive clusters within a separation of 6 arcmin. We show that PDFs in the denoised maps are not compromised by details of multiplicative biases and photometric redshift distributions, nor by shape measurement errors, and that the PDFs show stronger cosmological dependence compared to the noisy counterpart. We apply our denoising method to a part of the first-year HSC data to show that the observed mass distribution is statistically consistent with the prediction from the standard ΛCDM model. © 2021 The Author(s) Published by Oxford University Press on behalf of Royal Astronomical Society.","Cosmology: observations; Gravitational lensing: weak; Large-scale structure of Universe; Methods: data analysis",,,,,,"National Science Foundation, NSF: AST-1238877

National Aeronautics and Space Administration, NASA: NNX08AR22G

Princeton University

Johns Hopkins University, JHU

University of Maryland, UMD

University of Hawai'i, UH

Smithsonian Astrophysical Observatory, SAO

Space Telescope Science Institute, STScI

University of Edinburgh, ED

Queen's University Belfast, QUB

Durham University

Japan Society for the Promotion of Science, KAKEN

Ministry of Education, Culture, Sports, Science and Technology, Monbusho: 18H04358, 19K14767

Japan Science and Technology Agency, JST: JPMJCR1414

Max-Planck-Gesellschaft, MPG

University of Tokyo

National Astronomical Observatory of Japan, NAOJ

High Energy Accelerator Research Organization, KEK

Malaysia Toray Science Foundation, MTSF

Eötvös Loránd Tudományegyetem, ELTE

Leibniz-Institut für Astrophysik Potsdam, AIP: JP17K14273, JP19H00677, JP20317829","This work was in part supported by Grant-in-Aid for Scientific Research on Innovative Areas from the MEXT KAKENHI Grant Number (18H04358, 19K14767), and by Japan Science and Technology Agency CREST Grant Number JPMJCR1414 and AIP Acceleration Research Grant Number JP20317829. This work was also supported by JSPS KAKENHI Grant Numbers JP17K14273 and JP19H00677. Numerical computations presented in this paper were in part carried out on the general-purpose PC farm at Center for Computational Astrophysics, CfCA, of National Astronomical Observatory of Japan. The HSC collaboration includes the astronomical communities of Japan and Taiwan and Princeton University. The HSC instrumentation and software were developed by the National Astronomical Observatory of Japan (NAOJ), the Kavli Institute for the Physics and Mathematics of the Universe (Kavli IPMU), the University of Tokyo, the High Energy Accelerator Research Organization (KEK), the Academia Sinica Institute for Astronomy and Astrophysics in Taiwan (ASIAA), and PrincetonUniversity. Fundingwas contributed by the FIRST programme from Japanese Cabinet Office, the Ministry of Education, Culture, Sports, Science and Technology (MEXT), the Japan Society for the Promotion of Science (JSPS), Japan Science and Technology Agency (JST), the Malasiya Toray Science Foundation, NAOJ, Kavli IPMU, KEK, ASIAA, and Princeton University. This paper makes use of software developed for the Vera C. Rubin Observatory. We thank the LSST Project for making their code available as free software at http://dm.lsst.org. The Pan-STARRS1 Surveys (PS1) have been made possible through contributions of the Institute for Astronomy, the University of Hawaii, the Pan-STARRS Project Office, the Max-Planck Society and its participating institutes, the Max Planck Institute for Astronomy, Heidelberg and the Max Planck Institute for Extraterrestrial Physics, Garching, The Johns Hopkins University, Durham University, the University of Edinburgh, Queen's University Belfast, the Harvard-Smithsonian Center for Astrophysics, the Las Cumbres Observatory Global Telescope Network Incorporated, the National Central University of Taiwan, the Space Telescope Science Institute, the National Aeronautics and Space Administration under Grant Number NNX08AR22G issued through the Planetary Science Division of the NASA Science Mission Directorate, the National Science Foundation under Grant Number AST-1238877, the University of Maryland, and Eotvos LorandUniversity (ELTE) and the LosAlamos National Laboratory. Based [in part] on data collected at the Subaru Telescope and retrieved from the HSC data archive system, which is operated by Subaru Telescope and Astronomy Data Center at National Astronomical Observatory of Japan.",,"Adami, C., (2018) Astron. Astrophys, 620, p. A5; Ade, P. A. R., (2016) Astron. Astrophys, 594, p. A13; Aihara, H., (2018) PASJ, 70, p. S4; Ba, S., Myers, R. W., Brenneman, A. W., (2015) Technometrics, 57, p. 479; Bartelmann, M., Schneider, P., (2001) Phys. Rep, 340, p. 291; Becker, M. R., (2013) MNRAS, 435, p. 115; Behroozi, P. S., Wechsler, R. H., Wu, H.-Y., (2013) ApJ, 762, p. 109; Bernstein, G. M., Jarvis, M., (2002) AJ, 123, p. 583; Brock, A., Donahue, J., Simonyan, K., (2018), preprint (arXiv:1809.11096); Castro, T., Quartin, M., Giocoli, C., Borgani, S., Dolag, K., (2018) MNRAS, 478, p. 1305; Chang, C., (2018) MNRAS, 475, p. 3165; Clowe, D., Gonzalez, A., Markevitch, M., (2004) ApJ, 604, p. 596; Coulton, W. R., Liu, J., Madhavacheril, M. S., Böhm, V., Spergel, D. N., (2019) Journal of Cosmology and Astroparticle Physics, p. 043; Coupon, J., Czakon, N., Bosch, J., Komiyama, Y., Medezinski, E., Miyazaki, S., Oguri, M., (2018) PASJ, 70, p. S7; Crocce, M., Pueblas, S., Scoccimarro, R., (2006) MNRAS, 373, p. 369; Dietrich, J. P., Hartlap, J., (2010) MNRAS, 402, p. 1049; Fan, Z., Shan, H., Liu, J., (2010) ApJ, 719, p. 1408; Furusawa, H., (2018) PASJ, 70, p. S3; Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., (2014), preprint (arXiv:1406.2661); Gupta, A., Matilla, J. M. Z., Hsu, D., Haiman, Z., (2018) Phys. Rev., D97, p. 103515; Hamana, T., Mellier, Y., (2001) MNRAS, 327, p. 169; Hamana, T., Takada, M., Yoshida, N., (2004) MNRAS, 350, p. 893; Hikage, C., (2019) Publ. Astron. Soc. Jap, 71, p. 43; Hildebrandt, H., (2017) MNRAS, 465, p. 1454; Hinshaw, G., (2013) ApJS, 208, p. 19; Hirata, C. M., Seljak, U., (2003) MNRAS, 343, p. 459; Hirata, C. M., Mandelbaum, R., Ishak, M., Seljak, U., Nichol, R., Pimbblet, K. A., Ross, N. P., Wake, D., (2007) MNRAS, 381, p. 1197; Hu, W., White, M. J., (2001) ApJ, 554, p. 67; Huterer, D., Shafer, D. L., (2018) Rept. Prog. Phys, 81, p. 016901; Isola, P., Zhu, J.-Y., Zhou, T., Efros, A. A., (2016), preprint (arXiv:1611.07004); Jain, B., Seljak, U., White, S. D. M., (2000) ApJ, 530, p. 547; Jarvis, M., Bernstein, G., Jain, B., (2004) MNRAS, 352, p. 338; Jeffrey, N., Lanusse, F., Lahav, O., Starck, J.-L., (2020) MNRAS, 492, p. 5023; Kaiser, N., Squires, G., (1993) ApJ, 404, p. 441; Kingma, D. P., Ba, J., (2014), preprint (arXiv:1412.6980); Komiyama, Y., (2018) PASJ, 70, p. S2; Kratochvil, J. M., Haiman, Z., May, M., (2010) Phys. Rev. D, 81, p. 043519; Krause, E., Eifler, T., Blazek, J., (2016) MNRAS, 456, p. 207; Lewis, A., Challinor, A., Lasenby, A., (2000) ApJ, 538, p. 473; Lin, C.-A., Kilbinger, M., (2015) A&A, 576, p. A24; Liu, J., Madhavacheril, M. S., (2019) Phys. Rev, D99, p. 083508; Mandelbaum, R., (2018) PASJ, 70, p. S25; Mandelbaum, R., (2018) MNRAS, 481, p. 3170; Marques, G. A., Liu, J., Matilla, J. M. Z., Haiman, Z., Bernui, A., Novaes, C. P., (2019) JCAP, 1906, p. 019; Matsubara, T., Jain, B., (2001) ApJ, 552, p. L89; Miyazaki, S., (2015) ApJ, 807, p. 22; Miyazaki, S., (2018) PASJ, 70, p. S1; Moriwaki, K., Shirasaki, M., Yoshida, N., (2020) AJ, 906, p. 5; Murata, R., (2019) PASJ, 71, p. 107; Nishimichi, T., (2009) Publ. Astron. Soc. Jap, 61, p. 321; Nishimichi, T., (2019) ApJ, 884, p. 29; Oguri, M., (2018) Publ. Astron. Soc. Jap, 70, p. S20; Oguri, M., (2014) MNRAS, 444, p. 147; Osato, K., Shirasaki, M., Yoshida, N., (2015) ApJ, 806, p. 186; Pen, U.-L., Zhang, T.-J., van Waerbeke, L., Mellier, Y., Zhang, P.-J., Dubinski, J., (2003) ApJ, 592, p. 664; Petri, A., Liu, J., Haiman, Z., May, M., Hui, L., Kratochvil, J. M., (2015) Phys. Rev. D, 91, p. 103511; Press, W. H., Teukolsky, S. A., Vetterling, W. T., Flannery, B. P., (1992) Numerical recipes in FORTRAN, , The art of scientific computing; Remy, B., Lanusse, F., Ramzi, Z., Liu, J., Jeffrey, N., Starck, J.-L., (2020), preprint (arXiv:2011.08271); Ribli, D., Pataki, B. Á., Csabai, I., (2019) Nat. Astron, 3, p. 93; Ronneberger, O., Fischer, P., Brox, T., (2015), preprint (arXiv:1505.04597); Sato, J., Takada, M., Jing, Y. P., Futamase, T., (2001) ApJ, 551, p. L5; Sato, M., Hamana, T., Takahashi, R., Takada, M., Yoshida, N., Matsubara, T., Sugiyama, N., (2009) ApJ, 701, p. 945; Schmelzle, J., Lucchi, A., Kacprzak, T., Amara, A., Sgier, R., Réfrégier, A., Hofmann, T., (2017), preprint (arXiv:1707.05167); Schneider, P., (1996) MNRAS, 283, p. 837; Schneider, P., vanWaerbeke, L., Kilbinger, M., Mellier, Y., (2002) A&A, 396, p. 1; Seitz, C., Schneider, P., (1995) A&A, 297, p. 287; Shirasaki, M., (2017) MNRAS, 465, p. 1974; Shirasaki, M., Yoshida, N., (2014) ApJ, 786, p. 43; Shirasaki, M., Yoshida, N., Hamana, T., (2013) ApJ, 774, p. 111; Shirasaki, M., Hamana, T., Yoshida, N., (2015) MNRAS, 453, p. 3043; Shirasaki, M., Nishimichi, T., Li, B., Higuchi, Y., (2017) MNRAS, 466, p. 2402; Shirasaki, M., Takada, M., Miyatake, H., Takahashi, R., Hamana, T., Nishimichi, T., Murata, R., (2017) MNRAS, 470, p. 3476; Shirasaki, M., Hamana, T., Takada, M., Takahashi, R., Miyatake, H., (2019) MNRAS, 486, p. 52; Shirasaki, M., Yoshida, N., Ikeda, S., (2019) Phys. Rev, D100, p. 043527; Springel, V., (2005) MNRAS, 364, p. 1105; Takada, M., Jain, B., (2003) ApJ, 583, p. L49; Takahashi, R., Hamana, T., Shirasaki, M., Namikawa, T., Nishimichi, T., Osato, K., Shiroyama, K., (2017) ApJ, 850, p. 24; Tanaka, M., (2018) PASJ, 70, p. S9; Taruya, A., Takada, M., Hamana, T., Kayo, I., Futamase, T., (2002) ApJ, 571, p. 638; Troxel, M. A., (2018) MNRAS, 479, p. 4998; Troxel, M. A., (2018) Phys. Rev, D98, p. 043528; Troxel, M. A., Ishak, M., (2014) Phys. Rept, 558, p. 1; Tyson, J. A., Wenk, R. A., Valdes, F., (1990) ApJ, 349, p. L1; Valageas, P., Nishimichi, T., (2011) A&A, 527, p. A87; Vikram, V., (2015) Phys. Rev, D92, p. 022006; Wang, S., Haiman, Z., May, M., (2009) ApJ, 691, p. 547; Zaldarriaga, M., Scoccimarro, R., (2003) ApJ, 584, p. 559; Zorrilla Matilla, J. M., Sharma, M., Hsu, D., Haiman, Z., (2020) Physical Review D, 102, p. 123506","Shirasaki, M.; National Astronomical Observatory of JapanJapan; email: masato.shirasaki@nao.ac.jp",,,"Oxford University Press",,,,,00358711,,MNRAA,,"English","Mon. Not. R. Astron. Soc.",Article,"Final","All Open Access, Green",Scopus,2-s2.0-85107747502
"Lowney B., Lokmer I., O'Brien G.S., Bean C.J.","57209499174;23051107300;7102971135;7006261220;","Pre-migration diffraction separation using generative adversarial networks",2021,"Geophysical Prospecting","69","5",,"949","967",,,"10.1111/1365-2478.13086","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85104424556&doi=10.1111%2f1365-2478.13086&partnerID=40&md5=d5770222fdf1406d196078736fdbeac5","School of Earth Science, University College Dublin, Belfield, Dublin  D04 V1W8, Ireland; Irish Centre for Research in Applied Geoscience, University College Dublin, Belfield, Dublin  D04 V1W8, Ireland; Applied Geophysics and Technology, Tullow Oil Ltd., Dublin, D18 NH10, Ireland; School of Cosmic Physics, Dublin Institute of Advanced Studies, Dublin, D02 Y006, Ireland","Lowney, B., School of Earth Science, University College Dublin, Belfield, Dublin  D04 V1W8, Ireland, Irish Centre for Research in Applied Geoscience, University College Dublin, Belfield, Dublin  D04 V1W8, Ireland; Lokmer, I., School of Earth Science, University College Dublin, Belfield, Dublin  D04 V1W8, Ireland, Irish Centre for Research in Applied Geoscience, University College Dublin, Belfield, Dublin  D04 V1W8, Ireland; O'Brien, G.S., School of Earth Science, University College Dublin, Belfield, Dublin  D04 V1W8, Ireland, Applied Geophysics and Technology, Tullow Oil Ltd., Dublin, D18 NH10, Ireland; Bean, C.J., School of Cosmic Physics, Dublin Institute of Advanced Studies, Dublin, D02 Y006, Ireland","Diffraction imaging is the process of separating diffraction events from the seismic wavefield and imaging them independently, highlighting subsurface discontinuities. While there are many analytic-based methods for diffraction imaging which use kinematic, dynamic or both, properties of the diffracted wavefield, they can be slow and require parameterization. Here, we propose an image-to-image generative adversarial network to automatically separate diffraction events on pre-migrated seismic data in a fraction of the time of conventional methods. To train the generative adversarial network, plane-wave destruction was applied to a range of synthetic and real images from field data to create training data. These training data were screened and any areas where the plane-wave destruction did not perform well, such as synclines and areas of complex dip, were removed to prevent bias in the neural network. A total of 14,132 screened images were used to train the final generative adversarial network. The trained network has been applied across several geologically distinct field datasets, including a 3D example. Here, generative adversarial network separation is shown to be comparable to a benchmark separation created with plane-wave destruction, and up to 12 times faster. This demonstrates the clear potential in generative adversarial networks for fast and accurate diffraction separation. © 2021 European Association of Geoscientists & Engineers","Data processing; Imaging; Seismics","Elastic waves; Seismology; Separation; Adversarial networks; Conventional methods; Diffraction imaging; Plane wave destructions; Real images; Seismic datas; Seismic wavefields; Training data; Diffraction; artificial neural network; data processing; data set; image analysis; seismic data; seismic migration; seismic wave; wave diffraction; wave field",,,,,"Science Foundation Ireland, SFI: 13/RC/2092

European Regional Development Fund, ERDF","This research has emanated from research supported in part by a research grant from Science Foundation Ireland (SFI) under grant number 13/RC/2092 and is co‐funded under the European Regional Development Fund by PIPCO RSG and its member companies. The authors extend their gratitude to Tullow Oil and the Petroleum Affairs Division of Ireland for providing field data used in training. The authors would also like to thank Song Hou, Henning Hoeber and Ewa Kaszycka of CGG for their discussions on neural networks and diffractions. Finally, the authors would like to thank Shearwater for providing an academic license for Shearwater Reveal, which was used in this study.",,"Aharchaou, M., Baumstein, A., Deep learning-based artificial bandwidth extension: training on ultrasparse OBN to enhance towed-streamer FWI (2020) The Leading Edge, 39, pp. 718-726; Alotaibi, A., Deep generative adversarial networks for image-to-image translation: a review (2020) Symmetry, 12, pp. 1705-1731; Basheer, I., Hajmeer, M., Artificial neural networks: fundamentals, computing, design, and application (2000) Journal of Microbiological Methods, 43, pp. 3-31; Berkovitch, A., Belfer, I., Hassin, Y., Landa, E., Diffraction imaging by multifocusing (2009) Geophysics, 74 (6), pp. WCA75-WCA81; Bonnefoy-Claudet, S., Cotton, F., Bard, P., The nature of noise wavefield and its applications for site effects studies: a literature review (2006) Earth-Science Reviews, 79, pp. 205-227; Brownlee, J., (2019) How to develop a Pix2Pix GAN for image-to-image translation, , http://www.machinelearningmastery.com/how-to-develop-a-pix2pix-gan-for-image-to-image-translation, Accessed December 1, 2020; Cai, L., Gao, H., Ji, S., Multi-stage variational auto-encoders for coarse-to-fine image generation (2019), pp. 630-638; Chen, Z., Fomel, S., Lu, W., Accelerated plane-wave destruction (2013) Geophysics, 78 (1), pp. V1-V9; Claerbout, J., (1985) Fundamentals of Geophysical Data Processing with Applications to Petroleum Prospecting, , Hoboken, NJ, Blackwell Scientific Publications; Decker, L., Janson, X., Fomel, S., Carbonate reservoir characterization using seismic diffraction imaging (2015) Interpretation, 3 (1), pp. SF21-SF30; Decker, L., Klokov, A., Fomel, S., Comparison of seismic diffraction imaging techniques: plane-wave destruction versus apex destruction (2013) 83rd SEG Annual International Meeting, pp. 4054-4059. , Houston, TX, Expanded Abstracts; Dell, S., Gajewski, D., Common-reflection-surface-based workflow for diffraction imaging (2011) Geophysics, 76 (5), pp. S187-S195; Fehler, M., Keliher, P., (2011) SEAM Phase I: Challenge of Subsalt Imaging in Tertiary Basins, with Emphasis on Deepwater Gulf of Mexico, , Tulsa, OK, Society of Exploration Geophysicists; Fomel, S., Applications of plane-wave destruction filters (2002) Geophysics, 67 (6), pp. 1946-1960; Fomel, S., Landa, E., Taner, M., Poststack velocity analysis by separation and imaging of seismic diffractions (2007) Geophysics, 72 (6), pp. U89-U94; Fomel, S., Sava, P., Vlad, I., Liu, Y., Bashkardin, V., Madagascar: open-source software project for multidimensional data analysis and reproducible computational experiments (2013) Journal of Open Research Software, 1 (1); Gelius, L., Asgedom, E., Diffraction-limited imaging and beyond – the concept of super resolution (2011) Geophysical Prospecting, 59, pp. 400-421; Goodfellow, I., NIPS 2016 tutorial: generative adversarial networks (2016) Neural information Processing Systems 2016, Barcelona, Spain, pp. 1-57; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Generative adversarial nets (2014) Neural Information Processing Systems 2014, Montreal, Canada, pp. 2672-2680; Han, C., Murao, K., Sato, S., Nakayama, H., Learning more with less: GAN-based medical image augmentation (2019) ACM International Conference on Information and Knowledge Management, , Beijing; Harlan, W., Claerbout, J., Rocca, F., Signal/noise separation and velocity estimation (1984) Geophysics, 49 (11), pp. 1869-1880; Ho, Y., Wookey, S., The real-world-weight cross-entropy loss function: modeling the costs of mislabeling (2019) IEEE Access, 8, pp. 4806-4813; Isola, P., Zhu, J., Zhou, T., Efros, A., Image-to-image translation with conditional adversarial networks (2017) IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125-1134. , Manchester; JafarGandomi, A., Bukola, O., Refaar, R., Hoeber, H., Specular imaging of converted wave data and its AVO impact (2019) 81st EAGE Conference and Exhibition, pp. 1-5. , London, Expanded Abstracts; Kahng, M., Thorat, N., Chau, D.H.P., Viégas, F.B., Wattenberg, M., GAN lab: understanding complex deep generative models using interactive visual experimentation (2018) IEEE Transactions on Visualization and Computer Graphics, 25, pp. 1-11; Kanasewich, E., Phadke, S., Imaging discontinuities on seismic sections (1988) Geophysics, 53 (3), pp. 334-345; Kennett, B., (2001) The Seismic Wavefield: Introduction and Theoretical Development (Vol. 1), , Cambridge, Cambridge University Press; Khaidukov, V., Landa, E., Moser, T., Diffraction imaging by focusing-defocusing: an outlook on seismic superresolution (2004) Geophysics, 69 (6), pp. 1478-1490; Klokov, A., Baina, R., Landa, E., Separation and imaging of seismic diffractions in dip-angle domain (2010) 72nd EAGE Conference and Exhibition, , Barcelona, Spain, Expanded Abstracts, p; Klokov, A., Fomel, S., Separation and imaging of seismic diffractions using migrated dip-angle gathers (2012) Geophysics, 77 (6), pp. S131-S143; Knerr, S., Personnaz, L., Dreyfus, G., Single-layer learning revisited: a stepwise procedure for building and training a neural network (1990) Neurocomputing, pp. 41-50. , Soulié, F.F., Hérault, J., (Eds.), Heidelberg, Springer; Kozlov, E., Barasky, N., Korolev, E., Antonenko, A., Koshchuk, E., Imaging scattering objects masked by specular reflections (2004) 74th SEG Annual International Meeting, pp. 1131-1134. , Denver, CO, Expanded Abstracts; Krey, T., The significance of diffraction in the investigation of faults (1952) Geophysics, 17 (4), pp. 843-858; Landa, E., (2007) Beyond Conventional Seismic Imaging, , Houten, The Netherlands, European Association of Geoscientists and Engineers; Landa, E., Fomel, S., Moser, T., Path-integral seismic imaging (2006) Geophysical Prospecting, 54, pp. 491-503; Landa, E., Fomel, S., Reshef, M., Separation, imaging, and velocity analysis of seismic diffractions using migrated dip-angle gathers (2008) 78th SEG Annual International Meeting, pp. 2176-2180. , Las Vegas, NV; Li, R., Liu, W., Yang, L., Sun, S., Hu, W., Zhang, F., DeepUNet: a deep fully convolutional network for pixel-level sea-land segmentation (2018) IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11, pp. 3954-3962; Lian, S., Luo, Z., Zhong, Z., Lin, X., Su, S., Li, S., Attention guided U-Net for accurate iris segmentation (2018) Journal of Visual Communication, 56, pp. 296-304; Lowney, B., Lokmer, I., O'Brien, G.S., Bean, C.J., Direct diffraction separation by deep learning on pre-migrated seismic data (2020) 1st EAGE Annual Conference Online, pp. 1-5. , Houten, The Netherlands, European Association of Geoscientists and Engineers; Maciel, S., Biloti, R., Detection of diffractions in seismic sections using Support Vector Classifiers (2014) 74th SEG Annual International Conference, pp. 4816-4820. , Denver, CO, Expanded Abstracts; Martini, F., Bean, C.J., Dolan, S., Marsan, D., Seismic image quality beneath strongly scattering structures and implications for lower crustal imaging (2001) Geophysical Journal International, 145, pp. 423-435; Miller, J., Agena, W., Haines, S., Hart, P., (2016) Processing of multi-channel seismic reflection data acquired in 2013 for seismic investigation of gas hydrates in the Gulf of Mexico, , Reston, VA, U.S. Geological Survey, Open-File Report; Moser, T.J., Howard, C., Diffraction imaging in depth (2008) Geophysical Prospecting, 56, pp. 627-641; Nguyen, T., Le, T., Vu, H., Phung, D., Dual discriminator generative adversarial nets (2017) Advances in Neural Information Processing Systems, 1, pp. 2670-2680; O'Brien, G.S., Common image gather conditioning using cycle generative adversarial networks (2020) Geophysical Prospecting, 68, pp. 1758-1770; Oliveira, D., Ferreira, R., Silva, R., Brazil, E., Interpolating seismic data with conditional generative adversarial networks (2018) IEEE Geoscience and Remote Sensing Letters, 15, pp. 1952-1956; Popovici, A., Sturzu, I., Moser, T., High-resolution diffraction imaging of small-scale fractures in shale and carbonate reservoirs (2015) Unconventional Resources Technology Conference, San Antonio, TX, pp. 1121-1129; Radford, A., Metz, L., Chintala, S., Unsupervised representation learning with deep convolutional generative adversarial networks (2016) International Conference on Learning Representations, , San Juan, Puerto Rico; Reshef, M., Landa, E., Post-stack velocity analysis in the dip-angle domain using diffractions (2009) Geophysical Prospecting, 57, pp. 811-821; Ronneberger, O., Fischer, P., Brox, T., U-Net: convolutional networks for biomedical image segmentation (2015) International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234-241. , Boston, MA; Sankesara, H., (2019) UNet: introducing symmetry in segmentation, , https://towardsdatascience.com/u-net-b229b32b4a71, Accessed September 16, 2020; Schwarz, B., An introduction to seismic diffraction (2019) Advances in Geophysics: Recent Advances in Seismology, pp. 1-64. , C. Schmelzbach, (Ed.), New York, Academic Press; Schwarz, B., Coherent wavefield subtraction for diffraction separation (2019) Geophysics, 84 (3), p. V157. , -V168; Serfaty, Y., Itan, L., Levy, R., Koren, Z., Application of deep learning along directional image gathers for high-definition classification of subsurface features (2018) 80th EAGE Conference and Exhibition, Copenhagen, Denmark, Expanded Abstracts, pp. 1-5; Shekhar, A., (2019) What are L1 and L2 loss functions?, , https://afteracademy.com/blog/what-are-l1-and-l2-loss-functions, Accessed October 3, 2020; Sun, J., Slang, S., Elboth, T., Larsen Greiner, T., McDonald, S., Gelius, L.J., Attenuation of marine seismic interference noise employing a customized U-Net (2020) Geophysical Prospecting, 68, pp. 845-871; Tiwari, Y., Rasool, A., Hajela, G., Machine learning with generative adversarial networks (2020) Second International Conference on Inventive Resarch in Computing Applications (ICIRCA), pp. 543-548. , Expanded Abstracts; Tran, N., Tran, V., Nguyen, B., Yang, L., Self-supervised GAN: analysis and improvement with multi-class minimax game (2019) Advances in Neural Information Processing Systems, 32, pp. 13253-13264; Triezenberg, P.J., Hart, P.E., Childs, J.R., (2016) National Archive of Marine Seismic Surveys (NAMSS): A USGS data website of marine seismic reflection data within the U.S. Exclusive Economic Zone (EEZ), , Reston, VA, U.S. Geological Survey, s.l. U.S. Geological Survey Data Release; Trorey, A.W., A simple theory for seismic diffractions (1970) Geophysics, 35 (5), pp. 762-784; Tschannen, V., Ettrich, N., Delescluse, M., Keuper, J., Detection of point scatterers using diffraction imaging and deep learning (2020) Geophysical Prospecting, 68, pp. 830-844; Xu, R., Zhou, Z., Zhang, W., Yu, Y., Face transfer with generative adversarial network (2017) Conference on Computer Vision and Pattern Recognition, , Manchester, Expanded Abstracts; Yuan, Y., Si, X., Zheng, Y., Ground roll attenuation using generative adversarial networks (2020) Geophysics, 85 (4), pp. IJA-Z18; Zhang, T., Fu, H., Zhao, Y., Cheng, J., Guo, M., Gu, Z., SkrGAN: sketching-rendering unconditional generative adversarial networks for medical imaging synthesis (2019) International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 777-785. , Lima, Peru, Expanded Abstracts","Lowney, B.; School of Earth Science, Ireland; email: brydon.lowney@ucdconnect.ie",,,"Blackwell Publishing Ltd",,,,,00168025,,GPPRA,,"English","Geophys. Prospect.",Article,"Final","All Open Access, Hybrid Gold",Scopus,2-s2.0-85104424556
"Zhu J., Meng L., Wu W., Choi D., Ni J.","56136158700;57219023059;57218601347;25821918900;7201636285;","Generative adversarial network-based atmospheric scattering model for image dehazing",2021,"Digital Communications and Networks","7","2",,"178","186",,1,"10.1016/j.dcan.2020.08.003","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85091072068&doi=10.1016%2fj.dcan.2020.08.003&partnerID=40&md5=23ffe55f194dfa079301e4cd99170d03","College of Internet of Things Engineering, Hohai University, Changzhou, Jiangsu  213022, China; Jiangsu Provincial Collaborative Innovation Center of World Water Valley and Water Ecological Civilization, Nanjing, Jiangsu  211100, China; Chosun University, Gwangju, 61452, South Korea","Zhu, J., College of Internet of Things Engineering, Hohai University, Changzhou, Jiangsu  213022, China, Jiangsu Provincial Collaborative Innovation Center of World Water Valley and Water Ecological Civilization, Nanjing, Jiangsu  211100, China; Meng, L., College of Internet of Things Engineering, Hohai University, Changzhou, Jiangsu  213022, China; Wu, W., College of Internet of Things Engineering, Hohai University, Changzhou, Jiangsu  213022, China; Choi, D., Chosun University, Gwangju, 61452, South Korea; Ni, J., College of Internet of Things Engineering, Hohai University, Changzhou, Jiangsu  213022, China, Jiangsu Provincial Collaborative Innovation Center of World Water Valley and Water Ecological Civilization, Nanjing, Jiangsu  211100, China","This paper presents a trainable Generative Adversarial Network (GAN)-based end-to-end system for image dehazing, which is named the DehazeGAN. DehazeGAN can be used for edge computing-based applications, such as roadside monitoring. It adopts two networks: one is generator (G), and the other is discriminator (D). The G adopts the U-Net architecture, whose layers are particularly designed to incorporate the atmospheric scattering model of image dehazing. By using a reformulated atmospheric scattering model, the weights of the generator network are initialized by the coarse transmission map, and the biases are adaptively adjusted by using the previous round's trained weights. Since the details may be blurry after the fog is removed, the contrast loss is added to enhance the visibility actively. Aside from the typical GAN adversarial loss, the pixel-wise Mean Square Error (MSE) loss, the contrast loss and the dark channel loss are introduced into the generator loss function. Extensive experiments on benchmark images, the results of which are compared with those of several state-of-the-art methods, demonstrate that the proposed DehazeGAN performs better and is more effective. © 2020 Chongqing University of Posts and Telecommunications","Atmospheric scattering model; Contrast loss; Dehazing; Edge computing applications",,,,,,"National Research Foundation of Korea, NRF

Kementerian Pendidikan Malaysia, KPM: NRF-2018R1D1A1B07043331","This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (grant number NRF-2018R1D1A1B07043331 ).",,"Fattal, R., Single image dehazing (2008) ACM Trans. Graph., pp. 1-9; Mithun, N.C., Howlader, T., Rahman, S.M.M., Video-based tracking of vehicles using multiple time-spatial images (2016) Expert Syst Appl, pp. 17-31; Tian, B., Li, Y., Li, B., Wen, D., Rear-view vehicle detection and tracking by combining multiple parts for complex urban surveillance (2013) IEEE Trans. Intell. Transport. Syst., pp. 597-606; Negru, M., Nedevschi, S., Peter, R.I., Exponential contrast restoration in fog conditions for driving assistance (2015) IEEE Trans. Intell. Transport. Syst., pp. 2257-2268; Su, X., Yu, H.F., Kim, W.S., Choi, C., Choi, D.M., Interference cancellation for non-orthogonal multiple access used in future wireless mobile networks (2016) EURASIP J. Wirel. Commun. Netw., 2016, p. 231; Su, X., Fan, K., Shi, W., Privacy-preserving distributed data fusion based on attribute protection (2019) IEEE Trans. Industrial Info., 15 (10), pp. 5765-5777; Park, H., Park, D., Han, D.K., Ko, H., Single image haze removal using novel estimation of atmospheric light and transmission (2014) IEEE Int. Confer. Image Proc. (ICIP), pp. 4502-4506; Wang, J.-B., NingHe, Lu-LuZhang, KeLu, Single image dehazing with a physical model and dark channel prior (2015) Neurocomputing, pp. 718-728; Tan, R.T., Visibility in bad weather from a single image (2008) CVPR, pp. 1-8; He, K., Sun, J., Tang, X., Single image haze removal using dark channel prior (2011) IEEE Trans. Pattern Anal. Mach. Intell., pp. 2341-2353; Cai, B., Xu, X., Jia, K., Qing, C., Tao, D., DehazeNet: an end-to-end system for single image haze removal (2016) IEEE Trans. Image Process., pp. 5187-5198; Ren, W., Liu, S., Zhang, H., Pan, J., Cao, X., Yang, M.H., Single image dehazing via multi-scale convolutional neural networks (2016) ECCV, pp. 154-169; Wang, A., Wang, W., Liu, J., Gu, N., AIPNet: image-to-image single image dehazing with atmospheric illumination prior (2018) IEEE Trans. Image Process., pp. 381-393; Li, B., Peng, X., Wang, Z., AOD-net: all-in-one dehazing network (2017) IEEE Int. Confer. Comput. Vision (ICCV), pp. 4770-4778; Wu, W., Zhu, J., Su, X., DHGAN: generative adversarial network with dark channel prior for single-image dehazing (2019) Concurrency Comput. Pract. Ex., pp. 1-11; Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Generative adversarial nets (2014) NIPS, pp. 1-9; Ronneberger, O., Fischer, P., Brox, T., U-Net: convolutional networks for biomedical image segmentation (2015) The 18th International Conference on Medical Image Computing and Computer Assisted Interventions, pp. 234-241; Dong, H., Yang, G., Liu, F., Mo, Y., Guo, Y., Automatic brain tumor detection and segmentation using U-Net based fully convolutional networks (2017) Proc 21st Annu Conf Med Image Understand Anal, pp. 506-517; McCartney, E.J., Optics of the Atmosphere: Scattering by Molecules and Particles (1976), Wiley; Nayar, S.K., Narasimhan, S.G., Vision in bad weather (1999) Proceedings of the 7th IEEE International Conference on Computer Vision, 2, pp. 820-827. , 20-27 Sept 1999 Kerkyra, Greece; Fan, X., Ye, S., Shi, P., An image dehazing algorithm based on improved atmospheric scattering model (2019) J. Computer-Aided Des. Comput. Graph., 31 (7), pp. 1148-1155; Chen, Z.H., Wang, Y., Zou, Y.X., Inverse atmospheric scattering modeling with convolutional neural networks for single image dehazing (2018) IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2626-2630; Chen, S., Chen, S., Guo, Z., Zuo, Y., Low-resolution palmprint image denoising by generative adversarial networks (2019) Neurocomputing, pp. 275-284; Wei, Z., Bai, H., Zhao, Y., Stage-GAN with semantic maps for large-scale image super-resolution (2019) Trans. Internet and Info. sys., 13 (8), pp. 3942-3961; Donghyeon, L., Sangheon, L., Hoseong, L., Kyujoong, L., Hyuk-Jae, L., Resolution-Preserving generative adversarial networks for image enhancement (2019) IEEE Access, pp. 110344-110357; Malav, R., Kim, A., Sahoo, S.R., Pandey, G., DHSGAN: an end to end dehazing network for fog and smoke (2018) 14th Asian Conference on Computer Vision (ACCV), pp. 593-608; Zhao, J.M., Zhang, J., Li, Z., Hwang, J.N., DD-CycleGAN: Unpaired Image Dehazing via Double-Discriminator Cycle-Consistent Generative Adversarial Network. Engineering Applications of Artificial Intelligence (2019), pp. 263-271; Wu, F.F., Li, Y.F., Han, J.W., Perceptual Image Dehazing Based on Generative Adversarial Learning. Advances in Multimedia Information Processing (2018), pp. 877-887; Murugadoss, R., Ramakrishnan, M., Universal approximation of nonlinear system predictions in sigmoid activation functions using artificial neural networks (2014) IEEE International Conference on Computational Intelligence and Computing Research; Zhang, Y.D., Hou, X.X., Chen, Y., Voxelwise Detection of Cerebral Microbleed in CADASIL Patients by Leaky Rectified Linear Unit and Early Stopping (2017), pp. 1-21. , Multimedia Tools and Applications; Zeiler, Matthew, D., Fergus, R., Visualizing and understanding convolutional networks (2014) Proceedings of Computer Vision-ECCV, pp. 818-833; Li, J., Skinner, K.A., Eustice, R.M., Matthew, J.R., WaterGAN: unsupervised generative network to enable real-time color correction of monocular underwater images (2017) IEEE Robotics and Automation Lett., pp. 387-394; Ioffe, S., Szegedy, C., Batch normalization: accelerating deep network training by reducing internal covariate shift (2015) Proceedings of the 32nd International Conference on Machine Learning, pp. 448-456. , Lille, France 6-11 July 2015; Ketkar, N., Stochastic Gradient Descent (2017), Deep Learning with Python. Apress; Li, B., Ren, W., Fu, D., Benchmarking single image dehazing and beyond (2018) IEEE Trans. Image Process., pp. 492-505; Scharstein, D., Hirschmller, H., Kitajima, Y., High-resolution stereo datasets with subpixel-accurate ground truth (2014) The 36th German Conference on Pattern Recognition, 2-5 Sep 2014, pp. 1-12. , Munster Germany; Guo, F., Research on Image Defogging, Effect Assessment and Application (2012), Central South University; Huang, K.Q., Wang, Q., Wu, Z.Y., Natural Color Image Enhancement and Evaluation Algorithm Based on Human Visual System (2006), pp. 52-63. , CVIU; Hautiere, N., Tarel, J.P., Aubert, D., Blind contrast enhancement assessment by gradient ratioing at visible edges (2011) Image Anal. Stereol., pp. 87-95","Choi, D.; Chosun UniversitySouth Korea; email: jdmcc@chosun.ac.kr",,,"Chongqing University of Posts and Telecommunications",,,,,24685925,,,,"English","Digit. Commun Netw.",Article,"Final","All Open Access, Gold",Scopus,2-s2.0-85091072068
"Jimenez F., Koepke A., Gregg M., Frey M.","57212862289;57194574982;57223427340;7201869650;","Generative adversarial network performance in low-dimensional settings",2021,"Journal of Research of the National Institute of Standards and Technology","126",,"126008","","",,,"10.6028/jres.126.008","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85105872676&doi=10.6028%2fjres.126.008&partnerID=40&md5=0b0fc34dd88fd13ef7ae388b8a033c5a","National Institute of Standards and Technology, Gaithersburg, MD  20899, United States; University of Colorado Boulder, Boulder, CO  80309, United States","Jimenez, F., National Institute of Standards and Technology, Gaithersburg, MD  20899, United States, University of Colorado Boulder, Boulder, CO  80309, United States; Koepke, A., National Institute of Standards and Technology, Gaithersburg, MD  20899, United States; Gregg, M., National Institute of Standards and Technology, Gaithersburg, MD  20899, United States; Frey, M., National Institute of Standards and Technology, Gaithersburg, MD  20899, United States","A generative adversarial network (GAN) is an artifcial neural network with a distinctive training architecture, designed to create examples that faithfully reproduce a target distribution. GANs have recently had particular success in applications involving high-dimensional distributions in areas such as image processing. Little work has been reported for low dimensions, where properties of GANs may be better identifed and understood. We studied GAN performance in simulated low-dimensional settings, allowing us to transparently assess effects of target distribution complexity and training data sample size on GAN performance in a simple experiment. This experiment revealed two important forms of GAN error, tail underflling and bridge bias, where the latter is analogous to the tunneling observed in high-dimensional GANs. © 2021 National Institute of Standards and Technology. All rights reserved.","Earth mover distance; Experiment protocol; Generative adversarial network; Mode tunneling; Modeling error; Target distribution complexity","Engineering; Research; Adversarial networks; Artifcial neural networks; High-dimensional; Low dimensional; Training data; Image processing",,,,,,,,"Jin, Y, Zhang, J, Li, M, Tian, Y, Zhu, H, Fang, Z, (2017) Towards the automatic anime characters creation with generative adversarial networks, , https://arxiv.org/abs/1708.05509, arXiv preprint arXiv:1708.05509; Goodfellow, I, Pouget-Abadi, J, Mirza, M, Xu, B, Warde-Farley, D, Ozair, S, Courville, A, Bengio, Y, Generative adversarial networks (2018) Advances in Neural Information Processing Systems, pp. 2672-2680. , eds Jordan MI, LeCun Y, Solla SA (MIT Press, Cambridge, MA); Creswell, A, White, T, Dumoulin, V, Arulkumaran, K, Sengupta, B, Bharath, A, Generative Adversarial Networks: An Overview (2018) IEEE Signal Processing Magazine, 35 (1), pp. 53-65. , https://doi.org/10.1109/MSP.2017.2765202; Putin, E, Asadulaev, A, Vanhaelen, Q, Ivanenkov, Y, Aladinskaya, A, Aliper, A, Zhavoronkov, A, Adversarial Threshold Neural Computer for Molecular de Novo Design (2018) Molecular Pharmaceutics, 15 (10), pp. 4386-4397. , https://doi.org/10.1021/acs.molpharmaceut.7b01137; Mustafa, M, Bard, D, Bhimji, W, Lukíc, Z, Al-Rfou, R, Kratochvil, JM, CosmoGAN: Creating high-fdelity weak lensing convergence maps using Generative Adversarial Networks (2019) Computational Astrophysics and Cosmology, 6 (1), pp. 1-13. , https://doi.org/10.1186/s40668-019-0029-9; Hwang, U, Choi, S, Lee, HB, Yoon, S, (2017) Adversarial training for disease prediction from electronic health records with missing data, , https://arxiv.org/abs/1711.04126, arXiv preprint arXiv:1711.04126; Radfor, A, Metz, L, Chintala, S, (2015) Unsupervised representation learning with deep convolutional generative adversarial networks, , https://arxiv.org/abs/1511.06434, arXiv preprint arXiv:1511.06434; Isola, P, Zhu, JY, Zhou, T, Efros, A, Image-to-image translation with conditional adversarial networks (2017) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125-1134. , https://arxiv.org/abs/1611.07004; Karras, T, Aila, T, Laine, S, Lehtinen, J, (2017) Progressive growing of GANs for improved quality, stability, and variation, , https://arxiv.org/abs/1710.10196, arXivpreprint arXiv:1710.10196; Lim, Steffen, Khan, Sams, Alessandro, Matteo, McFall, Kevin, Spatio-temporal super-resolution with photographic and depth data using GANs (2019) Proceedings of the 2019 ACM Southeast Conference, pp. 262-263. , https://doi.org/10.1145/3299815.3314482; Zhu, JY, Park, T, Isola, P, Efros, A, Unpaired image-to-image translation using cycle-consistent adversarial networks (2017) Proceedings of the IEEE International Conference on Computer Vision, pp. 2223-2232. , https://doi.org/10.1109/ICCV.2017.244; Bau, D, Zhu, JY, Strobelt, H, Zhou, B, Tenenbaum, JB, Freeman, WT, Torralba, A, (2018) GAN dissection: Visualizing and understanding generative adversarial networks, , https://arxiv.org/abs/1811.10597, arXiv preprint arXiv:1811.10597; Lala, S, Shady, M, Belyaeva, A, Liu, M, Evaluation of mode collapse in generative adversarial networks (2018) Proceedings of theIEEE High Performance Extreme Computing Conference, , http://www.ieee-hpec.org/2018/2018program/indexhtmfles/124.pdf; Theis, L, van den Oord, A, Bethge, M, (2015) A note on the evaluation of generative models, , https://arxiv.org/abs/1511.01844, arXiv preprint arXiv:1511.01844; Salimans, T, Goodfellow, I, Zaremba, W, Cheung, V, Radford, A, Chen, X, Improved Techniques for Training GANs (2016) Advancesin Neural Information Processing Systems, pp. 2234-2242. , eds Jordan MI LeCun Y, Solla SA (MIT Press, Cambridge, MA); Lee, JD, Simchowitz, M, Jordan, MI, Recht, B, Gradient descent only converges to minimizers (2016) Proceedings of Machine Learning Research, 49, pp. 1246-1257. , http://proceedings.mlr.press/v49/lee16.pdf; Arjovsky, M, Bottou, L, (2017) Towards principled methods for training generative adversarial networks, , https://arxiv.org/abs/1701.04862, arXiv preprintarXiv:1701.04862; Arora, S, Ge, R, Liang, Y, Ma, T, Zhang, Y, Generalization and equilibrium in generative adversarial nets (GANs) (2017) Proceedings of Machine Learning Research, 70, pp. 224-232. , http://proceedings.mlr.press/v70/arora17a/arora17a.pdf; LeCun, Y, Bottou, L, Bengio, Y, Haffner, P, Gradient-based learning applied to document recognition (1998) Proceedings of theIEEE, 86 (11), pp. 2278-2324. , https://doi.org/10.1109/5.726791; Arjovsky, M, Chintala, S, Bottou, L, Wasserstein generative adversarial networks (2017) Proceedings of Machine Learning Research, 70, pp. 214-223. , http://proceedings.mlr.press/v70/arjovsky17a/arjovsky17a.pdf; Nagarajan, V, Kolter, JZ, Gradient Descent GAN Optimization is locally stable (2017) Advances in Neural Information Processing Systems, pp. 5585-5595. , eds Jordan MI, LeCun Y, Solla SA (MIT Press, Cambridge, MA); Gulrajani, I, Ahmed, F, Arjovsky, M, Dumoulin, V, Courville, AC, Improved Training of Wasserstein GANs (2017) Advances inNeural Information Processing Systems, pp. 5767-5777. , eds Jordan MI, LeCun Y, Solla SA (MIT Press, Cambridge, MA); Sønderby, CK, Caballero, J, Theis, L, Shi, W, Husźar, F, (2016) Amortised map inference for image super-resolution, , https://arxiv.org/abs/1610.04490, arXiv preprintarXiv:1610.04490; Mescheder, L, Geiger, A, Nowozin, S, Which training methods for GANs do actually converge? (2018) International Conferenceon Machine Learning, pp. 3481-3490. , https://arxiv.org/abs/1801.04406; Dobrushin, RL, Prescribing a system of random variables by conditional distributions (1970) Theory of Probability & Its Applications, 15 (3), pp. 458-486. , https://doi.org/10.1137/1115049; Mescheder, L, Nowozin, S, Geiger, A, The Numerics of GANs (2017) Advances in Neural Information Processing Systems, pp. 1825-1835. , eds Jordan MI, LeCun Y, Solla SA (MIT Press, Cambridge, MA); Kingma, DP, Ba, J, (2014) Adam: A method for stochastic optimization, , https://arxiv.org/abs/1412.6980, arXiv preprint arXiv:1412.6980; Monge, G, Memoire sur la theorie des deblais et des remblais (1781), Histoire de l'Academie Royale des Sciences de Paris (De l'Imprimerie Royale, Paris, France); Peleg, S, Werman, M, Rom, H, A unifed approach to the change of resolution: Space and gray-gevel (1989) IEEE Transactions on Pattern Analysis and Machine Intelligence, 11 (7), pp. 739-742. , https://doi.org/10.1109/34.192468; Rubner, Y, Tomasi, C, Guibas, LJ, A metric for distributions with applications to image databases (1998) Sixth International Conference on Computer Vision, pp. 59-66. , https://doi.org/10.1109/ICCV.1998.710701; Grauman, K, Darrell, T, Fast contour matching using approximate earth mover's distance (2004) Proceedings of the 2004 IEEEComputer Society Conference on Computer Vision and Pattern Recognition, , https://doi.org/10.1109/CVPR.2004.1315035; Cha, SH, Comprehensive survey on distance/similarity measures between probability density functions (2007) International Journal of Mathematical Models and Methods in Applied Sciences, 1 (4), pp. 300-307. , https://www.naun.org/main/NAUN/ijmmas/mmmas-49.pdf; Borji, A, Pros and cons of GAN evaluation measures (2019) Computer Vision and Image Understanding, 179, pp. 41-65. , https://doi.org/10.1016/j.cviu.2018.10.009; Cuturi, M, Sinkhorn distances: Lightspeed computation of optimal transport (2013) Advances in Neural Information Processing Systems, pp. 2292-2300. , https://arxiv.org/abs/1306.0895, eds Jordan MI, LeCun Y, Solla SA (MIT Press, Cambridge, MA); Auricchio, G, Bassetti, F, Gualandi, S, Veneroni, M, Computing Kantorovich-Wasserstein distances on d-dimensional histograms using (d + 1)-partite graphs (2018) Advances in Neural Information Processing Systems, pp. 5793-5803. , https://arxiv.org/abs/1805.07416, eds Jordan MI, LeCun Y, Solla SA (MIT Press, Cambrdige, MA); Flamary, R, Courty, N, (2019) Pot python optimal transport library, , https://pythonot.github.io/; Marsland, S, (2015) Machine Learning: An Algorithmic Perspective, , (CRC Press, Boca Raton, FL); Kutner, MH, Nachtsheim, CJ, Neter, J, Li, W, (2005) Applied Linear Statistical Models, 5. , (McGraw-Hill Irwin, Boston, MA); Brock, A, Donahue, J, Simonyan, K, Large scale GAN training for high fdelity natural image synthesis (2018) International Conference on Learning Representations, , https://arxiv.org/abs/1809.11096; Mirza, M, Osindero, S, (2014) Conditional generative adversarial nets, , https://arxiv.org/abs/1411.178, arXiv preprint arXiv:1411.1784",,,,"National Institute of Standards and Technology",,,,,21657254,,JRITE,,"English","J Res Natl Inst Stand Technol",Article,"Final","All Open Access, Bronze",Scopus,2-s2.0-85105872676
"Li X., Rosman G., Gilitschenski I., Vasile C.-I., Decastro J.A., Karaman S., Rus D.","57225160098;6602873213;55415577300;22959157700;35974903900;24923242500;57218886083;","Vehicle Trajectory Prediction Using Generative Adversarial Network with Temporal Logic Syntax Tree Features",2021,"IEEE Robotics and Automation Letters","6","2","9366373","3459","3466",,,"10.1109/LRA.2021.3062807","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85102265507&doi=10.1109%2fLRA.2021.3062807&partnerID=40&md5=8150578347057ed946c2a7bd042fc6c8","Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology, Cambridge, MA, United States; Computer Science and Artificial Intelligence Lab, Toyota Research Institute, Cambridge, United States; Department of Mechanical Engineering and Mechanics, Lehigh University, Bethlehem, PA, United States; Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, United States","Li, X., Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology, Cambridge, MA, United States; Rosman, G., Computer Science and Artificial Intelligence Lab, Toyota Research Institute, Cambridge, United States; Gilitschenski, I., Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology, Cambridge, MA, United States; Vasile, C.-I., Department of Mechanical Engineering and Mechanics, Lehigh University, Bethlehem, PA, United States; Decastro, J.A., Computer Science and Artificial Intelligence Lab, Toyota Research Institute, Cambridge, United States; Karaman, S., Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, United States; Rus, D., Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology, Cambridge, MA, United States","In this work, we propose a novel approach for integrating rules into traffic agent trajectory prediction. Consideration of rules is important for understanding how people behave-yet, it cannot be assumed that rules are always followed. To address this challenge, we evaluate different approaches of integrating rules as inductive biases into deep learning-based prediction models. We propose a framework based on generative adversarial networks that uses tools from formal methods, namely signal temporal logic and syntax trees. This allows us to leverage information on rule obedience as features in neural networks and improves prediction accuracy without biasing towards lawful behavior. We evaluate our method on a real-world driving dataset and show improvement in performance over off-The-shelf predictors. © 2016 IEEE.","Autonomous-driving; prediction; temporal logic","Agricultural robots; Deep learning; Forecasting; Formal methods; Predictive analytics; Syntactics; Temporal logic; Trees (mathematics); Adversarial networks; Inductive bias; Prediction accuracy; Prediction model; Real-world drivings; Traffic agents; Trajectory prediction; Vehicle trajectory predictions; Computer circuits",,,,,"Toyota Research Institute, TRI","Manuscript received October 15, 2020; accepted February 11, 2021. Date of publication March 1, 2021; date of current version March 23, 2021. This letter was recommended for publication by Associate Editor P. Tokekar and Editor D. Popa upon evaluation of the reviewers’ comments. This work has been supported by the Toyota Research Institute (TRI). (Corresponding author: Xiao Li.) Xiao Li, Igor Gilitschenski, and Daniela Rus are with the Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, United States of America (e-mail: xi-aoli@mit.edu; igilitschenski@mit.edu; rus@csail.mit.edu).",,"Mozaffari, S., Al-Jarrah, O.Y., Dianati, M., Jennings, P., Mouzakitis, A., Deep learning-based vehicle behaviour prediction for autonomous driving applications: A review (2019) arXiv:1912 11676; Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., Alahi, A., Social GAN: Socially acceptable trajectories with generative adversarial networks (2018) Proc Ieee Conf. Comput. Vis. Pattern Recognit, pp. 2255-2264; Li, J., Ma, H., Zhang, Z., Tomizuka, M., Social-WaGDAT: Interactionaware trajectory prediction via wasserstein graph double-Attention network (2020) arXiv:2002 06241; Li, X., Ying, X., Chuah, M., GRIP : Enhanced graph-based interactionaware trajectory prediction for autonomous driving (2019) Proc Ieee Intell. Transp. Syst. Conf, pp. 3960-3966; Salzmann, T., Ivanovic, B., Chakravarty, P., Pavone, M., Trajectron : Multi-Agent generative trajectory forecasting with heterogeneous data for control (2020) arXiv:2001 03093; Deo, N., Trivedi, M.M., Convolutional social pooling for vehicle trajectory prediction (2018) Proc Ieee Conf. Comput. Vis. Pattern Recognit. Workshops, pp. 1468-1476; Huang, X., McGill, S.G., Williams, B.C., Fletcher, L., Rosman, G., Uncertainty-Aware driver trajectory prediction at urban intersections (2019) Proc Ieee Int. Conf. Robot. Autom, pp. 9718-9724; Cui, H., Multimodal trajectory predictions for autonomous driving using deep convolutional networks (2019) Proc Ieee Int. Conf. Robot. Autom, pp. 2090-2096; Makansi, O., Ilg, E., Cicek, O., Brox, T., Overcoming limitations of mixture density networks: A sampling and fitting framework for multimodal future prediction (2019) Proc Ieee Conf. Comput. Vis. Pattern Recognit, pp. 7144-7153. , Jun; Shalev-Shwartz, S., Shammah, S., Shashua, A., On a formal model of safe and scalable self-driving cars (2017) arXiv:1708 06374; White, J., (2020) Illinois Rules of the Road, , https://cyberdriveillinois.com/publications/pdf-publications/dsd-A112.pdf; Raman, V., Donzé, A., Maasoumy, M., Murray, R., Sangiovanni-Vincentelli, A., Seshia, S., Model predictive control with signal temporal logic specifications (2014) Proc. 53rd Ieee Conf. Decis. Control, pp. 81-87; Caesar, H., NuScenes: A multimodal dataset for autonomous driving (2020) Proc IEEE/CVF Conf. Comput. Vis. Pattern Recognit, pp. 11621-11631; Donzé, A., Maler, O., Robust satisfaction of temporal logic over realvalued signals (2010) Proc. Int. Conf. FormalModel. Anal. Timed Syst, pp. 92-106; Lee, N., Choi, W., Vernaza, P., Choy, C.B., Torr, P., Chandraker, M., DESIRE: Distant future prediction in dynamic scenes with interacting agents (2017) Proc Ieee Conf. Comput. Vis. Pattern Recognit, pp. 336-345; Sadeghian, A., Kosaraju, V., Sadeghian, A., Hirose, N., Savarese, S., Sophie: An attentive gan for predicting paths compliant to social and physical constraints (2019) Proc Ieee Conf. Comput. Vis. Pattern Recognit, pp. 1349-1358; Chandra, R., Forecasting trajectory and behavior of road-Agents using spectral clustering in graph-LSTMs (2020) Ieee Robot. Automat. Lett, 5 (3), pp. 4882-4890. , Jun; Wang, E., Cui, H., Yalamanchi, S., Moorthy, M., Chou, F.-C., Djuric, N., Improving movement predictions of traffic actors in bird?s-eye view models using gans and differentiable trajectory rasterization (2020) Proc. 26th Acm Sigkdd Int. Conf. Knowl. Discov. Data Mining, pp. 2340-2348; Xu, Z., Topcu, U., Transfer of temporal logic formulas in reinforcement learning (2019) Proc. Int. Joint Conf. Artif. Intell, 28, p. 4010; Kapoor, P., Balakrishnan, A., Deshmukh, J., Model-based reinforcement learning from signal temporal logic specifications (2020) arXiv:2011 04950; Hasanbeig, M., Kantaros, Y., Abate, A., Kroening, D., Pappas, G.J., Lee, I., Reinforcement learning for temporal logic control synthesis with probabilistic satisfaction guarantees (2019) Proc Ieee 58th Conf. Decis. Control (CDC, pp. 5338-5343; Innes, C., Ramamoorthy, S., Elaborating on learned demonstrations with temporal logic specifications (2020) arXiv:2002 00784; Leung, K., Aréchiga, N., Pavone, M., Back-propagation through signal temporal logic specifications: Infusing logical structure into gradientbased methods (2020) arXiv:2008 00097; Censi, A., Liability, ethics, and culture-Aware behavior specification using rulebooks (2019) Proc Ieee Int. Conf. Robot. Autom, pp. 8536-8542; Vasile, C., Tumova, J., Karaman, S., Belta, C., Rus, D., Minimumviolation scLTl motion planning for mobility-on-demand (2017) Proc Ieee Int. Conf. Robot. Autom, pp. 1481-1488; Aréchiga, N., Specifying safety of autonomous vehicles in signal temporal logic (2019) Proc Ieee Intell. Veh. Symp. (IV, pp. 58-63; Park, D., Noseworthy, M., Paul, R., Roy, S., Roy, N., Inferring task goals and constraints using bayesian nonparametric inverse reinforcement learning (2020) Proc. Conf. Robot. Learn, pp. 1005-1014; Bansal, M., Krizhevsky, A., Ogale, A.S., Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst (2019) arXiv:1812 03079; Ding, W., Shen, S., Online vehicle trajectory prediction using policy anticipation network and optimization-based context reasoning (2019) Proc Ieee Int. Conf. Robot. Autom, pp. 9610-9616; Leung, K., Aréchiga, N., Pavone, M., Backpropagation for parametric STL (2019) Proc Ieee Intell. Veh. Symp, pp. 185-192; Goodfellow, I.J., Generative adversarial nets (2014) Proc. Int. Conf. Neural Inf. Process. Syst; Cui, H., Multimodal trajectory predictions for autonomous driving using deep convolutional networks (2019) Proc Ieee Int. Conf. Robot. Autom, pp. 2090-2096; Deo, N., Trivedi, M.M., Multi-modal trajectory prediction of surrounding vehicles with maneuver based LSTMs (2018) Proc. Intell. Veh. Symp, 4, pp. 1179-1184; Sandler, M., Howard, A.G., Zhu, M., Zhmoginov, A., Chen, L.-C., MobileNetV2: Inverted residuals and linear bottlenecks (2018) Proc Ieee Conf. Comput. Vis. Pattern Recognit, pp. 4510-4520","Li, X.; Computer Science and Artificial Intelligence Lab, United States; email: xiaoli@mit.edu",,,"Institute of Electrical and Electronics Engineers Inc.",,,,,23773766,,,,"English","IEEE Robot. Autom.",Article,"Final","",Scopus,2-s2.0-85102265507
"Beguš G.","56602923700;","Local and non-local dependency learning and emergence of rule-like representations in speech data by deep convolutional generative adversarial networks",2021,"Computer Speech and Language",,,"101244","","",,,"10.1016/j.csl.2021.101244","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85111558154&doi=10.1016%2fj.csl.2021.101244&partnerID=40&md5=89a6b0336623246eb856ea95ef6d84e4","Department of Linguistics, University of California, Berkeley, United States","Beguš, G., Department of Linguistics, University of California, Berkeley, United States","This paper argues that training Generative Adversarial Networks (GANs) on local and non-local dependencies in speech data offers insights into how deep neural networks discretize continuous data and how symbolic-like rule-based morphophonological processes emerge in a deep convolutional architecture. Acquisition of speech has recently been modeled as a dependency between latent space and data generated by GANs in Beguš (2020b), who models learning of a simple local allophonic distribution. We extend this approach to test learning of local and non-local phonological processes that include approximations of morphological processes. We further parallel outputs of the model to results of a behavioral experiment where human subjects are trained on the data used for training the GAN network. Four main conclusions emerge: (i) the networks provide useful information for computational models of speech acquisition even if trained on a comparatively small dataset of an artificial grammar learning experiment; (ii) local processes are easier to learn than non-local processes, which matches both behavioral data in human subjects and typology in the world's languages. This paper also proposes (iii) how we can actively observe the network's progress in learning and explore the effect of training steps on learning representations by keeping latent space constant across different training steps. Finally, this paper shows that (iv) the network learns to encode the presence of a prefix with a single latent variable; by interpolating this variable, we can actively observe the operation of a non-local phonological process. The proposed technique for retrieving learning representations has general implications for our understanding of how GANs discretize continuous speech data and suggests that rule-like generalizations in the training data are represented as an interaction between variables in the network's latent space. © 2021 The Author(s)","Behavioral experiments; Learning biases; Machine learning; Morphology; Neural networks; Speech","Behavioral research; Convolution; Convolutional neural networks; Deep neural networks; Learning systems; Speech; Adversarial networks; Behavioral experiment; Computational model; Continuous speech; Grammar learning; Latent variable; Morphological process; Speech acquisition; Deep learning",,,,,"University of California Berkeley, UCB

University of Washington, UW","This research was funded by a grant to new faculty at the University of Washington and UC Berkeley as well as by Harvard Mind Brain Behavior and Department of Linguistics.",,"Adlam, B., Weill, C., Kapoor, A., Investigating under and overfitting in Wasserstein Generative Adversarial Networks (2019), 2019. In ICML Understanding and Improving Generalization in Deep Learning Workshop. arXiv 1910.14137v1; Alishahi, A., Barking, M., Chrupała, G., Encoding of phonology in a recurrent neural model of grounded speech (2017) Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL), pp. 368-378. , https://www.aclweb.org/anthology/K17-1037, Association for Computational Linguistics, Vancouver, Canada; Arjovsky, M., Chintala, S., Bottou, L., Wasserstein Generative Adversarial Networks (2017) Proceedings of the 34th International Conference on Machine Learning. Vol. 70 of Proceedings of Machine Learning Research. PMLR, International Convention Centre, Sydney, Australia, pp. 214-223. , http://proceedings.mlr.press/v70/arjovsky17a.html, Precup D. Teh Y.W; Bates, D., Mächler, M., Bolker, B., Walker, S., Fitting linear mixed-effects models using lme4 (2015) J. Stat. Softw., 67 (1), pp. 1-48. , https://osf.io/c96b2, Available at /; Becker, M., Levine, J., (2013), http://becker.phonologist.org/experigen, Experigen – an online experiment platform; Beguš, G., CiwGAN and fiwGAN: encoding information in acoustic data to model lexical learning with Generative Adversarial Networks (2021) Neural Netw., 139, pp. 305-325. , https://www.sciencedirect.com/science/article/pii/S0893608021001052; Beguš, G., 2020a. Distinguishing cognitive from historical influences in phonology. In: submitted ms., UC Berkeley; Beguš, G., Generative adversarial phonology: modeling unsupervised phonetic and phonological learning with neural networks (2020) Front. Artif. Intell., 3, p. 44. , https://www.frontiersin.org/article/10.3389/frai.2020.00044; Berent, I., The phonological mind (2013) Trends Cognit. Sci., 17 (7), pp. 319-327. , http://www.sciencedirect.com/science/article/pii/S1364661313001034; Boersma, P., Weenink, D., (2015), http://www.praat.org/, Praat: doing phonetics by computer [computer program]. version 5.4.06. Retrieved 21 February 2015 from; Bond, Z.S., Wilson, H.F., /s/ plus stop clusters in children's speech (1980) Phonetica, 37 (3), pp. 149-158. , https://www.karger.com/DOI/10.1159/000259988; Brownlee, J., Generative adversarial networks with python: deep learning generative models for image synthesis and image translation (2019) Mach. Learn. Mastery; Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P., InfoGAN: interpretable representation learning by information maximizing Generative Adversarial Nets (2016) Advances in Neural Information Processing Systems 29. Curran Associates, Inc., pp. 2172-2180. , http://papers.nips.cc/paper/6399-infogan-interpretable-representation-learning-by-information-maximizing-generative-adversarial-nets.pdf, Lee D.D. Sugiyama M. Luxburg U.V. Guyon I. Garnett R; Chomsky, N., Halle, M., The Sound Pattern of English (1968), Harper & Row New York; Chung, Y.-A., Tang, H., Glass, J., Vector-quantized autoregressive predictive coding (2020) Proc. Interspeech 2020, pp. 3760-3764; Donahue, C., Mc Auley, J.J., Puckette, M.S., Adversarial audio synthesis (2019) 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, pp. 1-16. , https://openreview.net/forum?id=ByMVTsR5KQ; Eloff, R., Nortje, A., van Niekerk, B., Govender, A., Nortje, L., Pretorius, A., Biljon, E., Kamper, H., Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks (2019) Proc. Interspeech, pp. 1103-1107; Finley, S., The privileged status of locality in consonant harmony (2011) J. Memory Lang., 65 (1), pp. 74-83. , http://www.sciencedirect.com/science/article/pii/S0749596X11000192; Finley, S., Testing the limits of long-distance learning: learning beyond a three-segment window (2012) Cognit. Sci., 36 (4), pp. 740-756. , https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1551-6709.2011.01227.x; Garofolo, J.S., Lamel, L., M Fisher, W., Fiscus, J.S., Pallett, D.L., Dahlgren, N., Zue, V., Timit acoustic-phonetic continuous speech corpus (1993) Ling. Data Consortium, 11; Gaskell, M., Hare, M., Marslen-Wilson, W.D., A connectionist model of phonological representation in speech perception (1995) Cognit. Sci., 19 (4), pp. 407-439. , http://www.sciencedirect.com/science/article/pii/0364021395900071; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Advances in Neural Information Processing Systems 27. Curran Associates, Inc., pp. 2672-2680. , http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf, Ghahramani Z. Welling M. Cortes C. Lawrence N.D. Weinberger K.Q; Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C., Improved training of Wasserstein GANs (2017) Advances in Neural Information Processing Systems 30. Curran Associates, Inc., pp. 5767-5777. , http://papers.nips.cc/paper/7159-improved-training-of-wasserstein-gans.pdf, Guyon I. Luxburg U.V. Bengio S. Wallach H. Fergus R. Vishwanathan S. Garnett R; Hansson, G.O., Consonant Harmony: long-Distance Interactions in Phonology (2010), University of California Press; Heinz, J., Learning long-distance phonotactics (2010) Linguist. Inquiry, 41 (4), pp. 623-661; van der Hulst, H., Discoverers of the phoneme (2013) The Oxford Handbook of the History of Linguistics, pp. 167-191. , Allan K. Oxford University Press; Kabak, B., Turkish vowel harmony (2011) The Blackwell Companion to Phonology, pp. 1-24. , https://onlinelibrary.wiley.com/doi/abs/10.1002/9781444335262.wbctp0118, van Oostendorp M. Ewen C.J. Hume E. Rice K. Wiley Blackwell Ch. 118; Legendre, G., Miyata, Y., Smolensky, P., Harmonic Grammar: A Formal Multi-Level Connectionist Theory of Linguistic Well-Formedness: Theoretical Foundations (1990) ICS Technical Report #90-5, , University of Colorado, Boulder; MacMahon, M.K.C., Orthography and the early history of phonetics (2013) The Oxford Handbook of the History of Linguistics, pp. 105-122. , https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780199585847.001.0001/oxfordhb-9780199585847-e-6, Allan K. Oxford University Press; Marcus, G.F., The Algebraic Mind (2001) Integrating Connectionism and Cognitive Science, , MIT press; Marcus, G.F., Vijayan, S., Bandi Rao, S., Vishton, P.M., Rule learning by seven-month-old infants (1999) Science, 283 (5398), pp. 77-80. , https://science.sciencemag.org/content/283/5398/77; McClelland, J.L., Elman, J.L., The trace model of speech perception (1986) Cognit. Psychol., 18 (1), pp. 1-86. , http://www.sciencedirect.com/science/article/pii/0010028586900150; McClelland, J.L., Rumelhart, D.E., Group, P.R., Parallel Distributed Processing: Explorations in the Microstructure of Cognition (1986), 2. , MIT Press Cambridge, MA; McMullin, K., Hansson, G., Inductive learning of locality relations in segmental phonology (2019) Lab. Phonol. J. Assoc. Lab. Phonol., 10 (1), p. 14; Plaut, D.C., Kello, C.T., The emergence of phonology from the interplay of speech comprehension and production: a distributed connectionist approach (1999) The Emergence of Language, pp. 381-415. , Lawrence Erlbaum Associates Publishers Mahwah, NJ, US; Prince, A., Smolensky, P., Optimality Theory: Constraint (1993) Interaction in Generative Grammar, , Rutgers University Center for Cognitive Science Blackwell, Malden, MA First published in, Tech. Rep. 2; Radford, A., Metz, L., Chintala, S., Unsupervised representation learning with Deep Convolutional Generative Adversarial Networks (2015), n 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4 2016; Räsänen, O., Nagamine, T., Mesgarani, N., analyzing distributional learning of phonemic categories in unsupervised deep neural networks (2016) CogSci. Annual Conference of the Cognitive Science Society, pp. 1757-1762. , https://pubmed.ncbi.nlm.nih.gov/29359204, Cognitive Science Society (U.S.). Conference; Rose, S., Walker, R., A typology of consonant agreement as correspondence (2004) Language, 80 (3), pp. 475-531. , http://www.jstor.org/stable/4489721; Rumelhart, D.E., McClelland, J.L., Group, P.R., Parallel distributed processing (1986) Explorations in the Microstructure of Cognition, 1. , MIT Press Cambridge, MA; van Schijndel, M., Mueller, A., Linzen, T., Quantity doesn't buy quality syntax with neural language models (2019) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5831-5837. , https://www.aclweb.org/anthology/D19-1592, Association for Computational Linguistics, Hong Kong, China; Shain, C., Elsner, M., Measuring the perceptual availability of phonological features during language acquisition using unsupervised binary stochastic autoencoders (2019) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (long and short papers), pp. 69-85. , https://www.aclweb.org/anthology/N19-1007; Simon, N., Friedman, J., Hastie, T., Tibshirani, R., Regularization paths for Cox's proportional hazards model via coordinate descent (2011) J. Stat. Softw., 39 (5), pp. 1-13. , http://www.jstatsoft.org/v39/i05/; Smolensky, P., Goldrick, M., Gradient symbolic representations in grammar: the case of French liaison (2016) Rutgers Optimality Archive 1552, Rutgers University, pp. 1-37; Smolensky, P., Rosen, E., Goldrick, M., Learning a gradient grammar of French liaison (2019) Proceedings of the 2019 Annual Meeting on Phonology, pp. 1-12; Team, R.C., R: a language and environment for statistical computing (2018) R Foundation for Statistical Computing, Vienna, Austria, , https://www.R-project.org/; van de Vijver, R., Baer-Henney, D., Developing biases (2014) Front. Psychol., 5, p. 634. , https://www.frontiersin.org/article/10.3389/fpsyg.2014.00634; White, J., Nevins, A., Polgárdi, K., Martin, A., Kager, R., Linzen, T., Peperkamp, S., van de Vijver, R., Preference for locality is affected by the prefix/suffix asymmetry (2018) NELS 48: Proceedings of the Forty-Eighth Annual Meeting of the North East Linguistic Society, pp. 207-220. , Hucklebridge S. Nelson M. GLSA; Wood, S.N., Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models (2011) J. R. Stat. Soc. (B), 73 (1), pp. 3-36",,,,"Academic Press",,,,,08852308,,CSPLE,,"English","Comput Speech Lang",Article,"Article in Press","All Open Access, Hybrid Gold, Green",Scopus,2-s2.0-85111558154
"Tschaepe M.","44561711000;","Pragmatic ethics for generative adversarial networks: Coupling, cyborgs, and machine learning",2021,"Contemporary Pragmatism","18","1",,"95","111",,,"10.1163/18758185-bja10005","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85107447175&doi=10.1163%2f18758185-bja10005&partnerID=40&md5=ff70ad7747f04ee4b6b96d6140d6d3cf","Division of Social Work, Behavioral and Political Science, Prairie View A&M University, Prairie View, TX, United States","Tschaepe, M., Division of Social Work, Behavioral and Political Science, Prairie View A&M University, Prairie View, TX, United States","This article addresses the need for adaptive ethical analysis within machine learning that accounts for emerging problems concerning social bias and generative adversarial networks (gan s). I use John Dewey’s criticisms of the reflex arc concept in psychology as a basis for understanding how these problems stem from human-gan interaction. By combining Dewey’s criticisms with Donna Haraway’s idea of cyborgs, Luciano Floridi’s concept of distributed morality, and Shaowen Bardzell’s recommendations for a feminist approach to human-computer interaction, I suggest a dynamic perspective from which to begin analyzing and solving issues of injustice evident in this particular domain of machine learning. © koninklijke brill nv, leiden, 2021.","Bias; Coupling; Ethics of technology; Generative adversarial networks; Machine learning",,,,,,,,,"Ananny, M., Toward an ethics of algorithms: Convening, observation, probability, and timeliness (2016) Science, Technology, & Human Values, 41 (1), pp. 93-117; Angwin, J., Larson, J., Mattu, S., Kirchner, L., Machine bias (2016) ProPublica, p. 2016. , May 23; Asaro, P.M., Ai ethics in predictive policing: From models of threat to an ethics of care (2019) IEEE Technology and Society Magazine, 38 (2), pp. 40-53; Bardzell, S., Feminist HCI: Taking stock and outlining an agenda for design (2010) Proceedings of the Sigchi Conference on Human Factors in Computing Systems, pp. 1301-1310; Bardzell, S., Bardzell, J., Towards a feminist hci methodology: Social science, feminism, and hci (2011) Proceedings of the Sigchi Conference on Human Factors in Computing Systems, pp. 675-684; Benjamin, R., (2019) Race after Technology: Abolitionist Tools for the New Jim Code, , John Wiley & Sons; Borning, A., Muller, M., Next steps for value sensitive design (2012) Proceedings of Thesigchi Conference on Human Factors in Computing Systems, pp. 1125-1134; Borowiec, S., AlphaGo seals 4-1 victory over Go grandmaster Lee Sedol (2016) The Guardian, 15; Brey, P.A.E., Anticipatory ethics for emerging technologies (2012) NanoEthics, 6 (1), pp. 1-13; Broussard, M., (2018) Artificial Unintelligence: How Computers Misunderstand the World, , mit Press; Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., Tsaneva-Atanasova, K., Artificial intelligence, bias and clinical safety (2019) BMJ Quality & Safety, 28 (3), pp. 231-237; Ciston, S., Intersectional AI is essential (2019) Journal of Science and Technology of the Arts, 11 (2), pp. 3-8; Clark, A., Memento’s revenge: The extended mind, extended (2010) The Extended Mind, pp. 43-66. , The mit Press; Clark, A., Chalmers, D., The extended mind (1998) Analysis, 58 (1), pp. 7-19; Dastin, J., Amazon Scraps Secret Ai Recruiting Tool That Showed Bias against Women, , https://www.reuters.com/article/us-amazoncom-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-thatshowed-bias-against-women-idUSKCN1MK08G, San Francisco, CA: Reuters. Retrieved on October 9 (2018): 2018; de Preester, H., Technology and the body: The (im) possibilities of re-embodiment (2011) Foundations of Science, 16 (2-3), pp. 119-137; Dewey, J., The reflex arc concept in psychology (1896) John Dewey: The Early Works, 1882–1898, 5, pp. 96-109. , repr. in Jo Ann Boydston (Carbondale, IL: Southern Illinois University Press, 1972); Dewey, J., Imagination and expression (1896) John Dewey: The Early Works, 1882–1898, 5, pp. 192-201. , repr. in Jo Ann Boydston (Carbondale, IL: Southern Illinois University Press, 1972); Floridi, L., Distributed morality in an information society (2013) Science and Engineering Ethics, 19 (3), pp. 727-743; Friedman, B., Hendry, D.G., (2019) Value Sensitive Design: Shaping Technology with Moral Imagination, , mit Press; Friedman, B., Kahn, P.H., Borning, A., Value sensitive design and information systems (2008) The Handbook of Information and Computer Ethics, pp. 69-101; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Advances in Neural Information Processing Systems, pp. 2672-2680; Goodley, D., Lawthom, R., Cole, K.R., Posthuman disability studies (2014) Subjectivity, 7 (4), pp. 342-361; Haefner, J., Modest_witness in the wire: Haraway, predictive algorithms, and online profiling (2019) Auto/Biography Studies, 34 (3), pp. 403-422. , a/b; Haraway, D.J., (1991) Simians, Cyborgs and Women: The Reinvention of Nature, , Free Association Books; Haussler, D., Quantifying inductive bias: Ai learning algorithms and Valiant’s learning framework (1988) Artificial Intelligence, 36 (2), pp. 177-221; Ihde, D., (2012) Technics and Praxis: A Philosophy of Technology, 24. , Springer Science & Business Media, 1979; Ihde, D., (1995) Postphenomenology: Essays in the Postmodern Context, , Northwestern University Press; Ihde, D., (2002) Bodies in Technology, , University of Minnesota Press; Jain, N., Manikonda, L., Hernandez, A.O., Sengupta, S., Kambhampati, S., (2018) Imagining An Engineer: On Gan-Based Data Augmentation Perpetuating Biases, , arXiv preprint; Johnson, M., Rohrer, T., We are live creatures: Embodiment, American pragmatism and the cognitive organism Body, Language and Mind, 1, pp. 17-54. , Tom Ziemke, Jordan Zlatev, and Roz Frank, eds, Embodiment. Berlin: Mouton de Gruyter; Keeling, K., Queer os (2014) Cinema Journal, 53 (2), pp. 152-157; Lillywhite, A., Wolbring, G., Coverage of ethics within the artificial intelligence and machine learning academic literature: The case of disabled people (2019) Assistive Technology, pp. 1-7; Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A., (2019) A Survey on Bias and Fairness in Machine Learning, , arXiv preprint; Merleau-Ponty, M., (1945) Phénoménologie De La Perception, , Gallimard; Müller, V.C., Ethics of artificial intelligence (2021) The Routledge Social Science Handbook of AI, pp. 1-20. , Anthony Elliott (Ed.). Routledge; Munnik, R., Donna Haraway: Cyborgs for earthly survival? (2001) American Philosophy of Technology: The Empirical Turn, pp. 95-119; Noble, S.U., (2018) Algorithms of Oppression: How Search Engines Reinforce Racism, , nyu Press; O’Neil, C., (2016) Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, , Broadway Books; Páez, A., The pragmatic turn in explainable artificial intelligence (xai) (2019) Minds and Machines, 29 (3), pp. 441-459; Richards, D.P., John dewey, nonhuman agency, and the possibility of a posthuman public (2019) Contemporary Pragmatism, 16 (4), pp. 366-395; Rudin, C., Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead (2019) Nature Machine Intelligence, 1 (5), pp. 206-215; Saltz, J.S., Dewar, N.I., Heckman, R., Key concepts for a data science ethics curriculum (2018) Proceedings of the 49thacm Technical Symposium on Computer Science Education, pp. 952-957; Sejnowski, T.J., (2018) The Deep Learning Revolution, , mit Press; Shook, J.R., Giordano, J., Neuroethics beyond normal (2016) Cambridge Quarterly of Healthcare Ethics: CQ: The International Journal of Healthcare Ethics Committees, 25 (1), pp. 121-140; Still, A., d’Inverno, M., A history of creativity for future ai research (2016) Proceedings of the 7th Computational Creativity Conference (Iccc 2016), , Universite Pierre et Marie Curie; Still, A., d’Inverno, M., Can machines Be artists? A Deweyan response in theory and practice (2019) Arts, 8 (1), p. 36. , Multidisciplinary Digital Publishing Institute; van de Vijver, F.J.R., Towards a theory of bias and equivalence (1998) Zuma Nachrichten, 3, pp. 41-65; Venturelli, A.N., Dewey on the reflex arc and the dawn of the dynamical approach to the study of cognition (2012) Pragmatism Today, 3, pp. 132-143; Wang, F.-Y., Zhang, J.J., Zheng, X., Wang, X., Yuan, Y., Dai, X., Zhang, J., Yang, L., Where does AlphaGo go: From Church-Turing thesis to AlphaGo thesis and beyond (2016) IEEE/CAA Journal of Automatica Sinica, 3 (2), pp. 113-120","Tschaepe, M.; Division of Social Work, United States; email: mdtschaepe@pvamu.edu",,,"Brill Academic Publishers",,,,,15723429,,,,"English","Contemp. Pragmatism",Article,"Final","",Scopus,2-s2.0-85107447175
"Zheng Y., Sui X., Jiang Y., Che T., Zhang S., Yang J., Li H.","23471581800;57193155178;57208296460;57207830976;13605200100;15039078800;57141098300;","SymReg-GAN: Symmetric Image Registration with Generative Adversarial Networks",2021,"IEEE Transactions on Pattern Analysis and Machine Intelligence",,,,"","",,,"10.1109/TPAMI.2021.3083543","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85107219035&doi=10.1109%2fTPAMI.2021.3083543&partnerID=40&md5=a7673133d0e75664ea45a471a5bfc3c4","School of Information Science and Engineering, Shandong Normal University, 47856 Jinan, Shandong, China, (e-mail: zhengyuanjie@gmail.com); School of Information Science and Engineering, Shandong Normal University, 47856 Jinan, Shandong, China, (e-mail: xiaodan.sui@qq.com); School of Information Science and Engineering, Shandong Normal University, 47856 Jinan, Shandong, China, (e-mail: yanyun.jiang@qq.com); School of Biological Science and Medical Engineering, Beihang University, 12633 Beijing, Beijing, China, (e-mail: tong.che@qq.com); SenseTime Research, SenseTime Research, Beijing, Beijing, China, (e-mail: zhangshaoting@sensetime.com); Inst. Image Proc & Pattern Recog, Shanghai Jiao Tong University, Shanghai, Shanghai, China, 200240 (e-mail: jieyang@sjtu.edu.cn); Electronic Engineering, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong, (e-mail: hsli@ee.cuhk.edu.hk)","Zheng, Y., School of Information Science and Engineering, Shandong Normal University, 47856 Jinan, Shandong, China, (e-mail: zhengyuanjie@gmail.com); Sui, X., School of Information Science and Engineering, Shandong Normal University, 47856 Jinan, Shandong, China, (e-mail: xiaodan.sui@qq.com); Jiang, Y., School of Information Science and Engineering, Shandong Normal University, 47856 Jinan, Shandong, China, (e-mail: yanyun.jiang@qq.com); Che, T., School of Biological Science and Medical Engineering, Beihang University, 12633 Beijing, Beijing, China, (e-mail: tong.che@qq.com); Zhang, S., SenseTime Research, SenseTime Research, Beijing, Beijing, China, (e-mail: zhangshaoting@sensetime.com); Yang, J., Inst. Image Proc & Pattern Recog, Shanghai Jiao Tong University, Shanghai, Shanghai, China, 200240 (e-mail: jieyang@sjtu.edu.cn); Li, H., Electronic Engineering, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong, (e-mail: hsli@ee.cuhk.edu.hk)","Symmetric image registration estimates bi-directional spatial transformations between images while enforcing an inverse-consistency. Its capability of eliminating bias introduced inevitably by generic single-directional image registration allows more precise analysis in different interdisciplinary applications of image registration, e.g. computational anatomy and shape analysis. However, most existing symmetric registration techniques especially for multimodal images are limited by low speed from the commonly-used iterative optimization, hardship in exploring inter-modality relations or high labor cost for labeling data. We propose SymReg-GAN to shatter these limits, which is a novel generative adversarial networks (GAN) based approach to symmetric image registration. We formulate symmetric registration of unimodal/multimodal images as a conditional GAN and train it with a semi-supervised strategy. The registration symmetry is realized by introducing a loss for encouraging that the cycle composed of the geometric transformation from one image to another and its reverse should bring an image back. The semi-supervised learning enables both the precious labeled data and large amounts of unlabeled data to be fully exploited. Experimental results from 6 public brain magnetic resonance imaging (MRI) datasets and 1 our own computed tomography (CT) & MRI dataset demonstrate the superiority of SymReg-GAN to several existing state-of-the-art methods. CCBY","Estimation; Generative adversarial networks; Generative adversarial networks; Generators; Image registration; Image resolution; Magnetic resonance imaging; Multimodal image registration; Symmetric registration; Training","Image registration; Inverse problems; Iterative methods; Magnetic resonance imaging; Mathematical transformations; Semi-supervised learning; Wages; Adversarial networks; Computational anatomy; Geometric transformations; Iterative Optimization; Multi-modal image; Registration techniques; Spatial transformation; State-of-the-art methods; Computerized tomography",,,,,,,,,,,,"IEEE Computer Society",,,,,01628828,,ITPID,,"English","IEEE Trans Pattern Anal Mach Intell",Article,"Article in Press","All Open Access, Bronze",Scopus,2-s2.0-85107219035
"Athey S., Imbens G.W., Metzger J., Munro E.","6603691303;6603956309;57219636031;57218855868;","Using Wasserstein Generative Adversarial Networks for the design of Monte Carlo simulations",2021,"Journal of Econometrics",,,,"","",,1,"10.1016/j.jeconom.2020.09.013","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85102991297&doi=10.1016%2fj.jeconom.2020.09.013&partnerID=40&md5=f7e13756a76c78a40ba46a6d0228c654","Graduate School of Business, Stanford University, United States; NBER, United States; Department of Economics, Stanford University, United States","Athey, S., Graduate School of Business, Stanford University, United States, NBER, United States; Imbens, G.W., Graduate School of Business, Stanford University, United States, NBER, United States, Department of Economics, Stanford University, United States; Metzger, J., Department of Economics, Stanford University, United States; Munro, E., Graduate School of Business, Stanford University, United States","When researchers develop new econometric methods it is common practice to compare the performance of the new methods to those of existing methods in Monte Carlo studies. The credibility of such Monte Carlo studies is often limited because of the discretion the researcher has in choosing the Monte Carlo designs reported. To improve the credibility we propose using a class of generative models that has recently been developed in the machine learning literature, termed Generative Adversarial Networks (GANs) which can be used to systematically generate artificial data that closely mimics existing datasets. Thus, in combination with existing real data sets, GANs can be used to limit the degrees of freedom in Monte Carlo study designs for the researcher, making any comparisons more convincing. In addition, if an applied researcher is concerned with the performance of a particular statistical method on a specific data set (beyond its theoretical properties in large samples), she can use such GANs to assess the performance of the proposed method, e.g. the coverage rate of confidence intervals or the bias of the estimator, using simulated data which closely resembles the exact setting of interest. To illustrate these methods we apply Wasserstein GANs (WGANs) to the estimation of average treatment effects. In this example, we find that (i) there is not a single estimator that outperforms the others in all three settings, so researchers should tailor their analytic approach to a given setting, (ii) systematic simulation studies can be helpful for selecting among competing methods in this situation, and (iii) the generated data closely resemble the actual data. © 2021 Elsevier B.V.",,"Degrees of freedom (mechanics); Intelligent systems; Learning systems; Adversarial networks; Artificial data; Generative model; Machine learning literature; Monte Carlo's simulation; Performance; Property; Real data sets; Specific data sets; Study design; Monte Carlo methods",,,,,"Office of Naval Research, ONR: N00014-17-1-2131, N00014-19-1-2468

Alfred P. Sloan Foundation","We are grateful for comments by participants in the conference in honor of Whitney Newey in April 2019, and especially for discussions with Whitney about econometrics over many years. Financial support from the Sloan Foundation, United States of America and the Office of Naval Research, United States of America under grants N00014-17-1-2131 and N00014-19-1-2468 is gratefully acknowledged. We also want to acknowledge exceptional research assistance by Cole Kissane and Carolin Thomas. Replication code is available at https://github.com/evanmunro/dswgan-paper .",,"Abadie, A., Cattaneo, M.D., Econometric methods for program evaluation (2018) Annu. Rev. Econ., 10, pp. 465-503; Abadie, A., Imbens, G.W., Large sample properties of matching estimators for average treatment effects (2006) Econometrica, 74 (1), pp. 235-267; Abadie, A., Imbens, G.W., Bias-corrected matching estimators for average treatment effects (2011) J. Bus. Econom. Statist., 29 (1), pp. 1-11; Advani, A., Kitagawa, T., Słoczynński, T., Mostly harmless simulations? using monte carlo studies for estimator selection (2019) J. Appl. Econometrics; Arjovsky, M., Bottou, L., Towards principled methods for training generative adversarial networks (2017), arXiv preprint; Arjovsky, M., Chintala, S., Bottou, L., Wasserstein gan (2017), arXiv preprint; Athey, S., Imbens, G.W., Wager, S., Approximate residual balancing: debiased inference of average treatment effects in high dimensions (2018) J. R. Stat. Soc. Ser. B Stat. Methodol., 80 (4), pp. 597-623; Athey, S., Tibshirani, J., Wager, S., Generalized random forests (2019) Ann. Statist., 47 (2), pp. 1148-1178; Belloni, A., Chernozhukov, V., Hansen, C., Inference on treatment effects after selection among high-dimensional controls (2014) Rev. Econom. Stud., 81 (2), pp. 608-650; Berndt, E.R., Hall, B., Hall, R.E., Hausman, J.A., Estimation and inference in nonlinear structural models (1974) Annals of Economic and Social Measurement, Volume 3, pp. 653-665. , NBER number 4; Bottou, L., Large-scale machine learning with stochastic gradient descent (2010) Proceedings of COMPSTAT’2010, pp. 177-186. , Springer; Chen, X., Shen, X., Sieve extremum estimates for weakly dependent data (1998) Econometrica, pp. 289-314; Chen, X., White, H., Improved rates and asymptotic normality for nonparametric neural network estimators (1999) IEEE Trans. Inform. Theory, 45 (2), pp. 682-691; Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., Double/debiased/neyman machine learning of treatment effects (2017) Amer. Econ. Rev., 107 (5), pp. 261-265; Chetverikov, D., Testing regression monotonicity in econometric models (2019) Econometric Theory, 35 (4), pp. 729-776; Chetverikov, D., Santos, A., Shaikh, A.M., The econometrics of shape restrictions (2018) Annu. Rev. Econ., 10, pp. 31-63; Choromanska, A., Henaff, M., Mathieu, M., Arous, G.B., LeCun, Y., The loss surfaces of multilayer networks (2015) Artificial Intelligence and Statistics, pp. 192-204; Crump, R.K., Joseph Hotz, V., Imbens, G.W., Mitnik, O.A., Dealing with limited overlap in estimation of average treatment effects (2009) Biometrika, pp. 187-199; Cuturi, M., Sinkhorn distances: Lightspeed computation of optimal transport (2013) Advances in Neural Information Processing Systems, pp. 2292-2300; Dehejia, R.H., Wahba, S., Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs (1999) J. Amer. Statist. Assoc., 94 (448), pp. 1053-1062; Dehejia, R.H., Wahba, S., Propensity score-matching methods for nonexperimental causal studies (2002) Rev. Econ. Stat., 84 (1), pp. 151-161; Efron, B., The Jackknife, the Bootstrap and Other Resampling Plans (1982), SIAM; Efron, B., Tibshirani, R.J., An Introduction to the Bootstrap, Vol. 57 (1994), Chapman & Hall/CRC; Farrell, M.H., Robust inference on average treatment effects with possibly more covariates than observations (2015) J. Econometrics, 189 (1), pp. 1-23; Farrell, M.H., Liang, T., Misra, S., Deep neural networks for estimation and inference: Application to causal effects and other semiparametric estimands (2018), arXiv preprint; Goodfellow, I., Bengio, Y., Courville, A., Deep Learning (2016), MIT press; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Advances in Neural Information Processing Systems, pp. 2672-2680; Gui, J., Sun, Z., Wen, Y., Tao, D., Ye, J., A review on generative adversarial networks: Algorithms, theory, and applications (2020), arXiv preprint; Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C., Improved training of wasserstein gans (2017) Advances in Neural Information Processing Systems, pp. 5767-5777; Hahn, J., On the role of the propensity score in efficient semiparametric estimation of average treatment effects (1998) Econometrica, pp. 315-331; Härdle, W., Applied Nonparametric Regression. Number 19 (1990), Cambridge university press; Heckman, J.J., Joseph Hotz, V., Choosing among alternative nonexperimental methods for estimating the impact of social programs: The case of manpower training (1989) J. Amer. Statist. Assoc., 84 (408), pp. 862-874; Hirano, K., Imbens, G.W., Ridder, G., Efficient estimation of average treatment effects using the estimated propensity score (2003) Econometrica, 71 (4), pp. 1161-1189; Huber, M., Lechner, M., Wunsch, C., The performance of estimators based on the propensity score (2013) J. Econometrics, 175 (1), pp. 1-21; Huszár, F., How (not) to train your generative model: Scheduled sampling, likelihood, adversary? (2015), arXiv preprint; Imbens, G., Nonparametric estimation of average treatment effects under exogeneity: A review (2004) Rev. Econ. Stat., pp. 1-29; Imbens, G.W., Rubin, D.B., Causal Inference in Statistics, Social, and Biomedical Sciences (2015), Cambridge University Press; Kaji, T., Manresa, E., Poulio, G., Artificial Intelligence for Structural Estimation: Technical Report (2019), New York University; Kingma, D.P., Ba, J., Adam: A method for stochastic optimization (2014), arXiv preprint; Knaus, M., Lechner, M., Strittmatter, A., Machine learning estimation of heterogeneous causal effects: Empirical monte carlo evidence (2018); Kocaoglu, M., Snyder, C., Dimakis, A.G., Vishwanath, S., Causalgan: Learning causal implicit generative models with adversarial training (2017), arXiv preprint; LaLonde, R.J., Evaluating the econometric evaluations of training programs with experimental data (1986) Amer. Econ. Rev., pp. 604-620; Lechner, M., Strittmatter, A., Practical procedures to deal with common support problems in matching estimation (2019) Econometric Rev., 38 (2), pp. 193-207; Lechner, M., Wunsch, C., Sensitivity of matching-based program evaluations to the availability of control variables (2013) Labour Econ., 21, pp. 111-121; Liang, T., On how well generative adversarial networks learn densities: Nonparametric and parametric results (2018), arXiv preprint; Liu, Y., Qin, Z., Wan, T., Luo, Z., Auto-painter: Cartoon image generation from sketch by using conditional wasserstein generative adversarial networks (2018) Neurocomputing, 311, pp. 78-87; Ma, X., Wang, J., Robust inference using inverse probability weighting (2010) J. Amer. Statist. Assoc., pp. 1-10; Mirza, M., Osindero, S., Conditional generative adversarial nets (2014), arXiv preprint; Odena, A., Olah, C., Shlens, J., Conditional image synthesis with auxiliary classifier gans (2017) Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2642-2651. , JMLR. org; Rosenbaum, P.R., Rubin, D.B., The central role of the propensity score in observational studies for causal effects (1983) Biometrika, 70 (1), pp. 41-55; Scharfstein, D.O., Rotnitzky, A., Robins, J.M., Comments and rejoinder (1999) J. Amer. Statist. Assoc., 94 (448), pp. 1121-1146; Schuler, A., Jung, K., Tibshirani, R., Hastie, T., Shah, N., Synth-validation: Selecting the best causal inference method for a given dataset (2017), arXiv preprint; Silverman, B.W., Density Estimation for Statistics and Data Analysis (2018), Routledge; Singh, S., Uppal, A., Li, B., Li, C.-L., Zaheer, M., Póczos, B., Nonparametric density estimation under adversarial losses (2018) Advances in Neural Information Processing Systems, pp. 10225-10236; Tsybakov, A.B., Introduction to Nonparametric Estimation (2008), Springer Science & Business Media; Wager, S., Athey, S., Estimation and inference of heterogeneous treatment effects using random forests (2018) J. Amer. Statist. Assoc., 113 (523), pp. 1228-1242; Wager, S., Wang, S., Liang, P.S., Dropout training as adaptive regularization (2013) Advances in Neural Information Processing Systems, pp. 351-359; Warde-Farley, D., Goodfellow, I.J., Courville, A., Bengio, Y., An empirical analysis of dropout in piecewise linear networks (2013), arXiv preprint; Wendling, T., Jung, K., Callahan, A., Schuler, A., Shah, N.H., Gallego, B., Comparing methods for estimation of heterogeneous treatment effects using observational data from health care databases (2018) Statist. Med., 37 (23), pp. 3309-3324; Xie, L., Lin, K., Wang, S., Wang, F., Zhou, J., Differentially private generative adversarial network (2018)","Athey, S.; Graduate School of Business, 655 Knight Way, United States; email: athey@stanford.edu",,,"Elsevier Ltd",,,,,03044076,,JECMB,,"English","J Econom",Article,"Article in Press","All Open Access, Green",Scopus,2-s2.0-85102991297
"Li X., Fang M., Li H.","55221177500;38061325500;57202907763;","Bias alleviating generative adversarial network for generalized zero-shot classification",2021,"Image and Vision Computing","105",,"104077","","",,1,"10.1016/j.imavis.2020.104077","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85097198764&doi=10.1016%2fj.imavis.2020.104077&partnerID=40&md5=db8733f24cdd7374b4777b34a54898b7","School of Computer Science and Technology, Xidian University710071, China","Li, X., School of Computer Science and Technology, Xidian University710071, China; Fang, M., School of Computer Science and Technology, Xidian University710071, China; Li, H., School of Computer Science and Technology, Xidian University710071, China","Generalized zero-shot classification is predicting the labels of the test images coming from seen or unseen classes. The task is difficult because of the bias problem, that is, unseen samples are easily to be misclassified to seen classes. Many methods have handled the problem by training a generative adversarial network (GAN) to generate fake samples. However, the GAN model trained with seen samples might not be appropriate for generating unseen samples. For dealing with this problem, we learn a bias alleviating generative adversarial network for generalized zero-shot classification by generating seen and unseen samples, simultaneously. We train the generator to generate more realistic unseen samples by adding semantic similarity and cluster center regularizations to alleviate the bias problem. The semantic similarity regularization is to restrict the relationships of the generated unseen visual prototypes and seen visual prototypes by their class prototypes to avoid the generated unseen samples similar to the seen samples. The cluster center regularization is to utilize the cluster property of target data to make the generated unseen visual prototypes near to the most similar cluster centers, generating realistic unseen samples. From the experiments, we can see the proposed method achieves promising results. © 2020 Elsevier B.V.","Cluster centers; Generalized zero shot classification; Generative adversarial network; Semantic relationships; Unseen visual prototypes","Computer applications; Electrical engineering; Adversarial networks; Bias problems; Cluster centers; Cluster property; Semantic similarity; Shot classification; Test images; Semantics",,,,,"1908085MF186

National Natural Science Foundation of China, NSFC: 61806155

China Postdoctoral Science Foundation: 2018M631125

Natural Science Foundation of Shaanxi Province: 2020GY-062, 2020JQ-323

Fundamental Research Funds for the Central Universities: XJS200303","This work is supported by National Natural Science Foundation of China under Grant no. 61806155 , China Postdoctoral Science Foundation funded project under Grant no. 2018M631125 , Fundamental Research Funds for the Central Universities under Grant no. XJS200303 , National Natural Science Foundation of shaanxi province (Grant No. 2020JQ-323 , 2020GY-062 ), Nature Science Foundation of Anhui Province under Grant no. 1908085MF186 .",,"Lampert, C.H., Nickisch, H., Harmeling, S., Attribute-based classification for zero-shot visual object categorization (2014) Pattern Analysis and Machine Intelligence, IEEE Transactions on, 36 (3), pp. 453-465; Farhadi, A., Endres, I., Hoiem, D., Forsyth, D., Describing objects by their attributes. In CVPR (2009) IEEE, pp. 1778-1785; Socher, R., Ganjoo, M., Manning, C.D., Ng, A., Zero-shot learning through cross-modal transfer (2013) NeurIPS, pp. 935-943; Elhoseiny, M., Saleh, B., Elgammal, A., Write a classifier: zero-shot learning using purely textual descriptions (2013) In ICCV, pp. 2584-2591; Changpinyo, S., Chao, W.-L., Sha, F., Predicting visual exemplars of unseen classes for zero-shot learning (2017) In ICCV, pp. 3476-3485; Liu, Z., Zhang, X., Zhu, Z., Zheng, S., Zhao, Y., Cheng, J., Convolutional prototype learning for zero-shot recognition (2020) Image and Vision Computing, page, , 103924; Yu, Y., Zhong Ji, Jichang Guo, and Zhongfei Zhang (2018), Zero-shot learning via latent space encoding. IEEE transactions on cybernetics; Xian, Y., Lampert, C.H., Schiele, B., Akata, Z., Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly (2018) IEEE Transactions on Pattern Analysis and Machine Intelligence; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) In NeurIPS, pp. 2672-2680; Li, J., Jing, M., Lu, K., Ding, Z., Zhu, L., Huang, Z., Leveraging the Invariant Side of Generative Zero-Shot Learning (2019), In CVPR June; Akata, Z., Perronnin, F., Harchaoui, Z., Cordelia Schmid. Attribute-based classification with label-embedding. In NIPS (2013) Workshop on Output Representation Learning, p. 2013; Romera-Paredes, B., Torr, P.H.S., An embarrassingly simple approach to zero-shot learning (2015) In ICML, pp. 2152-2161; Kodirov, E., Xiang, T., Gong, S., Semantic autoencoder for zero-shot learning (2017) In CVPR, pp. 3174-3183; Kodirov, E., Xiang, T., Zhenyong, F., Gong, S., Unsupervised domain adaptation for zero-shot learning (2015) In ICCV, pp. 2452-2460; Xu, X., Shen, F., Yang, Y., Zhang, D., Shen, H.T., Song, J., Matrix tri-factorization with manifold regularizations for zero-shot learning (2017) In CVPR, pages, 2007-2016; Ding, Z., Shao, M., Yun, F., Low-rank embedded ensemble semantic dictionary for zero-shot learning (2017) In CVPR, pp. 2050-2058; Shigeto, Y., Suzuki, I., Hara, K., Shimbo, M., Matsumoto, Y., Ridge regression, hubness, and zero-shot learning (2015) Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 135-151. , Springer; Li, X., Fang, M., Feng, D., Li, H., Wu, J., Learning unseen visual prototypes for zero-shot classification (2018) Knowl.-Based Syst., 160, pp. 176-187; Zhang, L., Xiang, T., Gong, S., Learning a deep embedding model for zero-shot learning (2017) CVPR, pp. 2021-2030; Yanwei, F., Hospedales, T.M., Xiang, T., Fu, Z., Gong, S., Transductive multi-view embedding for zero-shot recognition and annotation (2014) ECCV, pp. 584-599. , Springer; Xian, Y., Akata, Z., Sharma, G., Nguyen, Q., Hein, M., Schiele, B., Latent embeddings for zero-shot classification (2016) In CVPR, pp. 69-77; Gan, C., Yang, T., Gong, B., Learning attributes equals multi-source domain generalization (2016) In CVPR, pp. 87-97; Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome, A., Corrado, G.S., Dean, J., Zero-shot learning by convex combination of semantic embeddings (2014) ICLR, pp. 1-9; Changpinyo, S., Chao, W.-L., Gong, B., Sha, F., Synthesized classifiers for zero-shot learning (2016) In CVPR, pp. 5327-5336; Li, X., Fang, M., Wu, J., Zero-shot classification by transferring knowledge and preserving data structure (2017) Neurocomputing, 238, pp. 76-83; Chao, W.-L., Changpinyo, S., Gong, B., Sha, F., An empirical study and analysis of generalized zero-shot learning for object recognition in the wild (2016) ECCV, pp. 52-68. , Springer; Liu, S., Long, M., Wang, J., Jordan, M.I., Generalized zero-shot learning with deep calibration network (2018) NeurIPS, pp. 2005-2015; Zhang, F., Shi, G., Co-representation network for generalized zero-shot learning (2019) In ICML, pp. 7434-7443; Song, J., Shen, C., Yang, Y., Yang, L., Song, M., Transductive unbiased embedding for zero-shot learning (2018) In CVPR, pp. 1024-1033; (2019), Akanksha Paul, Narayanan C Krishnan, and Prateek Munjal. Semantically aligned bias reducing zero shot learning. In CVPR, pages 7056–7065; Atzmon, Y., Chechik, G., Adaptive confidence smoothing for generalized zero-shot learning (2019) In CVPR, pp. 11671-11680; (2018), Hongguang Zhang and Piotr Koniusz. Model selection for generalized zero-shot learning. In ECCV, pages 0–0; Schonfeld, E., Ebrahimi, S., Sinha, S., Darrell, T., Akata, Z., Generalized zero-and few-shot learning via aligned variational autoencoders (2019) In CVPR, pp. 8247-8255; Wang, D., Li, Y., Lin, Y., Zhuang, Y., Relational knowledge transfer for zero-shot learning (2016) In AAAI, pp. 2145-2151; Yang, L., Liu, L., Shen, F., Shao, L., Li, X., Zero-shot learning using synthesised unseen visual data with diffusion regularisation (2018) IEEE Trans. Pattern Anal. Mach. Intell., 40 (10), pp. 2498-2512; He, H., Wang, C., Yu, P.S., Wang, C.-D., Generative dual adversarial network for generalized zero-shot learning (2019) CVPR, pp. 801-810; Xian, Y., Lorenz, T., Schiele, B., Akata, Z., Feature generating networks for zero-shot learning (2018) In CVPR, pp. 5542-5551; Felix, R., Kumar, V.B.G., Reid, I., Carneiro, G., Multi-modal cycle-consistent generalized zero-shot learning (2018) ECCV, pp. 21-37; Zhu, Y., Elhoseiny, M., Liu, B., Peng, X., Elgammal, A., A generative adversarial approach for zero-shot learning from noisy texts (2018) In CVPR, pp. 4321-4330; (2019), Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-vaegan-d2: A feature generating framework for any-shot learning. In CVPR, pages 10275–10284; Ni, J., Zhang, S., Xie, H., Dual adversarial semantics-consistent network for generalized zero-shot learning (2019) In NeurIPS, pp. 6143-6154; (2014), Mehdi Mirza and Simon Osindero. Conditional Generative Adversarial Nets. arXiv preprint; Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C., Improved training of wasserstein gans (2017) NeurIPS, pp. 5767-5777; Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S., The Caltech-Ucsd Birds-200-2011 Dataset (2011); (2012), Genevieve Patterson and James Hays. Sun attribute database: discovering, annotating, and recognizing scene attributes. In CVPR, pages 2751–2758. IEEE; Li, J., Jing, M., Lu, K., Ding, Z., Zhu, L., Huang, Z., Leveraging the invariant side of generative zero-shot learning (2019) In CVPR, pp. 7402-7411; van der Maaten, L., Hinton, G., Visualizing data using t-sne (2008) J. Mach. Learn. Res., 9 (Nov), pp. 2579-2605","Fang, M.; School of Computer Science and Technology, China; email: mfang@mail.xidian.edu.cn",,,"Elsevier Ltd",,,,,02628856,,IVCOD,,"English","Image Vision Comput",Article,"Final","",Scopus,2-s2.0-85097198764
"Suh S., Lee H., Lukowicz P., Lee Y.O.","57206939487;57203779723;6701557151;35758850500;","CEGAN: Classification Enhancement Generative Adversarial Networks for unraveling data imbalance problems",2021,"Neural Networks","133",,,"69","86",,3,"10.1016/j.neunet.2020.10.004","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85094124687&doi=10.1016%2fj.neunet.2020.10.004&partnerID=40&md5=98b3fca2c8a1880a0c08adf4cee55c01","Smart Convergence Group, Korea Institute of Science and Technology Europe Forschungsgesellschaft mbH, Saarbrücken, 66123, Germany; Department of Computer Science, TU Kaiserslautern, Kaiserslautern, 67663, Germany; German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, 67663, Germany","Suh, S., Smart Convergence Group, Korea Institute of Science and Technology Europe Forschungsgesellschaft mbH, Saarbrücken, 66123, Germany, Department of Computer Science, TU Kaiserslautern, Kaiserslautern, 67663, Germany; Lee, H., Smart Convergence Group, Korea Institute of Science and Technology Europe Forschungsgesellschaft mbH, Saarbrücken, 66123, Germany; Lukowicz, P., Department of Computer Science, TU Kaiserslautern, Kaiserslautern, 67663, Germany, German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, 67663, Germany; Lee, Y.O., Smart Convergence Group, Korea Institute of Science and Technology Europe Forschungsgesellschaft mbH, Saarbrücken, 66123, Germany","The data imbalance problem in classification is a frequent but challenging task. In real-world datasets, numerous class distributions are imbalanced and the classification result under such condition reveals extreme bias in the majority data class. Recently, the potential of GAN as a data augmentation method on minority data has been studied. In this paper, we propose a classification enhancement generative adversarial networks (CEGAN) to enhance the quality of generated synthetic minority data and more importantly, to improve the prediction accuracy in data imbalanced condition. In addition, we propose an ambiguity reduction method using the generated synthetic minority data for the case of multiple similar classes that are degenerating the classification accuracy. The proposed method is demonstrated with five benchmark datasets. The results indicate that approximating the real data distribution using CEGAN improves the classification performance significantly in data imbalanced conditions compared with various standard data augmentation methods. © 2020 Elsevier Ltd","Ambiguous classes; Classification enhancement; Data augmentation; Generative adversarial networks; Imbalanced classification","Artificial intelligence; Cognitive systems; Adversarial networks; Benchmark datasets; Class distributions; Classification accuracy; Classification performance; Classification results; Prediction accuracy; Real-world datasets; Classification (of information); article; prediction; automated pattern recognition; classification; data analysis; human; procedures; Data Analysis; Humans; Neural Networks, Computer; Pattern Recognition, Automated",,,,,"12020","This research was supported by Korea Institute of Science and Technology Europe Institutional Program (Project No. 12020 ).",,"Arjovsky, M., Chintala, S., Bottou, L., (2017), Wasserstein generative adversarial networks. In International conference on machine learning (pp. 214–223); Arthur, D., Vassilvitskii, S., k-means++: The advantages of careful seeding (2007) Proceedings of the eighteenth annual ACM-SIAM symposium on discrete algorithms, pp. 1027-1035. , Society for Industrial and Applied Mathematics; Barua, S., Islam, M.M., Yao, X., Murase, K., MWMOTE–Majority weighted minority oversampling technique for imbalanced data set learning (2012) IEEE Transactions on Knowledge and Data Engineering, 26 (2), pp. 405-425; Beijbom, O., Edmunds, P.J., Kline, D.I., Mitchell, B.G., Kriegman, D., Automated annotation of coral reef survey images (2012) 2012 IEEE conference on computer vision and pattern recognition, pp. 1170-1177. , IEEE; Buda, M., Maki, A., Mazurowski, M.A., A systematic study of the class imbalance problem in convolutional neural networks (2018) Neural Networks, 106, pp. 249-259; Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P., SMOTE: synthetic minority over-sampling technique (2002) Journal of Artificial Intelligence Research, 16, pp. 321-357; Cohen, G., Afshar, S., Tapson, J., van Schaik, A., EMNIST: an extension of MNIST to handwritten letters (2017), arXiv preprint; Darlow, L.N., Crowley, E.J., Antoniou, A., Storkey, A.J., CINIC-10 is not imagenet or CIFAR-10 (2018), arXiv preprint; Douzas, G., Bacao, F., Effective data generation for imbalanced learning using conditional generative adversarial networks (2018) Expert Systems with Applications, 91, pp. 464-471; Gao, X., Deng, F., Yue, X., Data augmentation in fault diagnosis based on the wasserstein generative adversarial network with gradient penalty (2019) Neurocomputing; Glorot, X., Bengio, Y., (2010), Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics (pp. 249–256); Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Generative adversarial nets (2014) Advances in neural information processing systems, pp. 2672-2680; Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y., Maxout networks (2013), arXiv preprint; Graves, S., Asner, G., Martin, R., Anderson, C., Colgan, M., Kalantari, L., Tree species abundance predictions in a tropical agricultural landscape with a supervised classification model and imbalanced data (2016) Remote Sensing, 8 (2), p. 161; Grzymala-Busse, J.W., Goodwin, L.K., Grzymala-Busse, W.J., Zheng, X., An approach to imbalanced data sets based on changing rule strength (2004) Rough-neural computing, pp. 543-553. , Springer; Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C., Improved training of wasserstein gans (2017) Advances in neural information processing systems, pp. 5767-5777; Haixiang, G., Yijing, L., Shang, J., Mingyun, G., Yuanyue, H., Bing, G., Learning from class-imbalanced data: Review of methods and applications (2017) Expert Systems with Applications, 73, pp. 220-239; Han, H., Wang, W.-Y., Mao, B.-H., Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning (2005) International conference on intelligent computing, pp. 878-887. , Springer; He, H., Bai, Y., Garcia, E.A., Li, S., ADASYN: Adaptive synthetic sampling approach for imbalanced learning (2008) 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence), pp. 1322-1328. , IEEE; He, H., Garcia, E.A., Learning from imbalanced data (2008) IEEE Transactions on Knowledge & Data Engineering, (9), pp. 1263-1284; Holzinger, A., Machine learning for health informatics: state-of-the-art and future challenges (vol. 9605) (2016), Springer; Japkowicz, N., Stephen, S., The class imbalance problem: A systematic study (2002) Intelligent Data Analysis, 6 (5), pp. 429-449; Jeatrakul, P., Wong, K.W., Fung, C.C., Classification of imbalanced data by combining the complementary neural network and SMOTE algorithm (2010) International conference on neural information processing, pp. 152-159. , Springer; Johnson, B.A., Tateishi, R., Hoan, N.T., A hybrid pansharpening approach and multiscale object-based image analysis for mapping diseased pine and oak trees (2013) International Journal of Remote Sensing, 34 (20), pp. 6969-6982; Kingma, D.P., Ba, J., Adam: A method for stochastic optimization (2014), arXiv preprint; Krizhevsky, A., Hinton, G., Learning multiple layers of features from tiny images: Tech. rep. (2009), Citeseer; Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks (2012) Advances in neural information processing systems, pp. 1097-1105; Kubat, M., Holte, R.C., Matwin, S., Machine learning for the detection of oil spills in satellite radar images (1998) Machine Learning, 30 (2-3), pp. 195-215; Larsen, A.B.L., Sønderby, S.K., Larochelle, H., Winther, O., Autoencoding beyond pixels using a learned similarity metric (2015), arXiv preprint; LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., Gradient-based learning applied to document recognition (1998) Proceedings of the IEEE, 86 (11), pp. 2278-2324; LeCun, Y., Cortes, C., Burges, C., MNIST handwritten digit database (vol. 2) (2010), p. 18. , http://yann.lecun.com/exdb/mnist, AT&T Labs [Online]. Available:; Lee, Y.O., Jo, J., Hwang, J., Application of deep neural network and generative adversarial network to industrial maintenance: A case study of induction motor fault detection (2017) 2017 IEEE international conference on big data, pp. 3248-3253. , IEEE; Lee, M., Seok, J., Controllable generative adversarial network (2019) IEEE Access, 7, pp. 28158-28169; Liu, T., Li, G., The imbalanced data problem in the fault diagnosis of rolling bearing (2010) Computer Engineering & Science, 32 (5), pp. 150-153; Lu, X., Chen, M., Wu, J., Chan, P., A feature-partition and under-sampling based ensemble classifier for web spam detection (2015) International Journal of Machine Learning and Computing, 5 (6), p. 454; Lusa, L., Evaluation of smote for high-dimensional class-imbalanced microarray data (2012) 2012 11th International conference on machine learning and applications (vol. 2), pp. 89-94. , IEEE; Ma, K., Wu, Q., Wang, Z., Duanmu, Z., Yong, H., Li, H., Zhang, L., (2016), Group mad competition-a new methodology to compare objective image quality models. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1664–1673); Maaten, L.V.D., Hinton, G., Visualizing data using t-SNE (2008) Journal of Machine Learning Research, 9 (Nov), pp. 2579-2605; Mac Namee, B., Cunningham, P., Byrne, S., Corrigan, O.I., The problem of bias in training data in regression problems in medical decision support (2002) Artificial Intelligence in Medicine, 24 (1), pp. 51-70; Mariani, G., Scheidegger, F., Istrate, R., Bekas, C., Malossi, C., Bagan: Data augmentation with balancing gan (2018), arXiv preprint; Mirza, M., Osindero, S., Conditional generative adversarial nets (2014), arXiv preprint; Ng, W.W., Hu, J., Yeung, D.S., Yin, S., Roli, F., Diversified sensitivity-based undersampling for imbalance classification problems (2014) IEEE Transactions on Cybernetics, 45 (11), pp. 2402-2412; Nguyen, H.M., Cooper, E.W., Kamei, K., Borderline over-sampling for imbalanced data classification (2011) International Journal of Knowledge Engineering and Soft Data Paradigms, 3 (1), pp. 4-21; Odena, A., Olah, C., Shlens, J., Conditional image synthesis with auxiliary classifier gans (2017) Proceedings of the 34th international conference on machine learning (vol. 70), pp. 2642-2651. , JMLR. org; Qian, N., On the momentum term in gradient descent learning algorithms (1999) Neural Networks, 12 (1), pp. 145-151; Radford, A., Metz, L., Chintala, S., Unsupervised representation learning with deep convolutional generative adversarial networks (2015), arXiv preprint; Ramentol, E., Caballero, Y., Bello, R., Herrera, F., SMOTE-RSB*: a hybrid preprocessing approach based on oversampling and undersampling for high imbalanced data-sets using SMOTE and rough sets theory (2012) Knowledge and Information Systems, 33 (2), pp. 245-265; Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X., Improved techniques for training gans (2016) Advances in neural information processing systems, pp. 2234-2242; Schmidhuber, J., Generative adversarial networks are special cases of artificial curiosity (1990) and also closely related to predictability minimization (1991) (2020) Neural Networks, 127, pp. 58-66; Simonyan, K., Zisserman, A., Very deep convolutional networks for large-scale image recognition (2014), arXiv preprint; Suh, S., Lee, H., Jo, J., Lukowicz, P., Lee, Y.O., Generative oversampling method for imbalanced data on bearing fault detection and diagnosis (2019) Applied Sciences, 9 (4), p. 746; Van Horn, G., Mac Aodha, O., Song, Y., Shepard, A., Adam, H., Perona, P., The inaturalist challenge 2017 dataset (vol. 1) (2017), arXiv preprint; Wang, Z., Simoncelli, E.P., Bovik, A.C., Multiscale structural similarity for image quality assessment (2003) The thrity-seventh asilomar conference on signals, systems & computers (vol. 2), pp. 1398-1402. , IEEE; Wang, Q., Zhou, X., Wang, C., Liu, Z., Huang, J., Zhou, Y., WGAN-based synthetic minority over-sampling technique: Improving semantic fine-grained classification for lung nodules in CT images (2019) IEEE Access, 7, pp. 18450-18463; Wold, S., Esbensen, K., Geladi, P., Principal component analysis (1987) Chemometrics and Intelligent Laboratory Systems, 2 (1-3), pp. 37-52; Wong, S.C., Gatt, A., Stamatescu, V., McDonnell, M.D., Understanding data augmentation for classification: when to warp? (2016) 2016 International conference on digital image computing: techniques and applications, pp. 1-6. , IEEE; Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A., Sun database: Large-scale scene recognition from abbey to zoo (2010) 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. , IEEE; Xiao, H., Rasul, K., Vollgraf, R., Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms (2017), arXiv preprint; Xie, J., Qiu, Z., The effect of imbalanced data sets on LDA: A theoretical and empirical analysis (2007) Pattern Recognition, 40 (2), pp. 557-562; Zhao, X.-M., Li, X., Chen, L., Aihara, K., Protein classification with imbalanced data (2008) Proteins: Structure, function, and bioinformatics, 70 (4), pp. 1125-1132","Lee, Y.O.; Smart Convergence Group, Germany; email: yongoh.lee@kist-europe.de",,,"Elsevier Ltd",,,,,08936080,,NNETE,"33125919","English","Neural Netw.",Article,"Final","",Scopus,2-s2.0-85094124687
"Alonso-Monsalve S., Whitehead L.H.","57192663084;57171887200;","Image-Based Model Parameter Optimization Using Model-Assisted Generative Adversarial Networks",2020,"IEEE Transactions on Neural Networks and Learning Systems","31","12","9032341","5645","5650",,4,"10.1109/TNNLS.2020.2969327","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85095340177&doi=10.1109%2fTNNLS.2020.2969327&partnerID=40&md5=5c70247ce0c75f181f37c40784eea7b1","Department of Computer Science and Engineering, Universidad Carlos III de Madrid, Leganés, 1211, Spain; CERN, Geneva, 1211, Switzerland","Alonso-Monsalve, S., Department of Computer Science and Engineering, Universidad Carlos III de Madrid, Leganés, 1211, Spain; Whitehead, L.H., CERN, Geneva, 1211, Switzerland","We propose and demonstrate the use of a model-assisted generative adversarial network (GAN) to produce fake images that accurately match true images through the variation of the parameters of the model that describes the features of the images. The generator learns the model parameter values that produce fake images that best match the true images. Two case studies show excellent agreement between the generated best match parameters and the true parameters. The best match model parameter values can be used to retune the default simulation to minimize any bias when applying image recognition techniques to fake and true images. In the case of a real-world experiment, the true images are experimental data with unknown true model parameter values, and the fake images are produced by a simulation that takes the model parameters as input. The model-assisted GAN uses a convolutional neural network to emulate the simulation for all parameter values that, when trained, can be used as a conditional generator for fast fake-image production. © 2012 IEEE.","Fast simulation; generative adversarial networks (GANs); model-assisted GAN; parameter optimization","Image recognition; Adversarial networks; Best match; Case-studies; Image production; Image-based modeling; Model parameters; Real world experiment; Convolutional neural networks; article; convolutional neural network; image display; simulation",,,,,,,,"Goodfellow, I., Generative adversarial nets (2014) Proc. Adv. Neural Inf. Process. Syst., pp. 2672-2680; Mirza, M., Osindero, S., (2014) Conditional Generative Adversarial Nets, , https://arxiv.org/abs/1411.1784; Antipov, G., Baccouche, M., Dugelay, J.-L., (2017) Face Aging with Conditional Generative Adversarial Networks, , https://arxiv.org/abs/1702.01983; Wu, C., Memory replay GANs: Learning to generate new categories without forgetting (2018) Proc. Adv. Neural Inf. Process. Syst., pp. 5962-5972; Hu, Z., (2018) Deep Generative Models with Learnable Knowledge Constraints, , http://arxiv.org/abs/1806.09764; Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P., InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets (2016) Proc. Adv. Neural Inf. Process. Syst., pp. 2172-2180; Creswell, A., Bharath, A.A., Inverting the generator of a generative adversarial network (2019) IEEE Trans. Neural Netw. Learn. Syst., 30 (7), pp. 1967-1974. , Jul; Radovic, A., Machine learning at the energy and intensity frontiers of particle physics (2018) Nature, 560 (7716), pp. 41-48. , Aug; De Oliveira, L., Paganini, M., Nachman, B., Learning particle physics by example: Location-aware generative adversarial networks for physics synthesis (2017) Comput. Softw. Big Sci., 1 (1), p. 4. , Sep; Paganini, M., De Oliveira, L., Nachman, B., CaloGAN: Simulating 3D high energy particle showers in multilayer electromagnetic calorimeters with generative adversarial networks (2018) Phys. Rev. D, Part. Fields, 97 (1). , Jan; Paganini, M., De Oliveira, L., Nachman, B., Accelerating science with generative adversarial networks: An application to 3D particle showers in multilayer calorimeters (2018) Phys. Rev. Lett., 120 (4). , Jan; Schawinski, K., Zhang, C., Zhang, H., Fowler, L., Santhanam, G.K., Generative adversarial networks recover features in astrophysical images of galaxies beyond the deconvolution limit (2017) Monthly Notices Roy. Astronomical Soc., Lett., 467 (1), pp. L110-L114; Bromley, J., Signature verification using a 'siamese' time delay neural network (1993) Int. J. Pattern Recog. Artif. Intell., 7 (4), pp. 669-688; Chopra, S., Hadsell, R., LeCun, Y., Learning a similarity metric discriminatively, with application to face verification (2005) Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (CVPR), 1, pp. 539-546. , Jun; Taigman, Y., Yang, M., Ranzato, M., Wolf, L., DeepFace: Closing the gap to human-level performance in face verification (2014) Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 1701-1708. , Jun; Schroff, F., Kalenichenko, D., Philbin, J., (2015) FaceNet: A Unified Embedding for Face Recognition and Clustering, , http://arxiv.org/abs/1503.03832; Goodfellow, I., Bengio, Y., Courville, A., (2016) Deep Learning. Cambridge, , http://www.deeplearningbook.org, MA, USA: MIT Press; Radford, A., Metz, L., Chintala, S., (2015) Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, , https://arxiv.org/abs/1511.06434; Chollet, F., (2015), https://github.com/keras-team/keras; Abadi, M., Tensorflow: A system for large-scale machine learning (2016) Proc. Osdi, 16, pp. 265-283; Chintala, S., Denton, E., Arjovsky, M., Mathieu, M., (2016) How to Train a GAN? Tips and Tricks to Make GANs Work., , https://github.com/soumith/ganhacks; Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X., (2016) Improved Techniques for Training GANs, , https://arxiv.org/abs/1606.03498; Arjovsky, M., Bottou, L., (2017) Towards Principled Methods for Training Generative Adversarial Networks, , https://arxiv.org/abs/1701.04862; Bottou, L., Curtis, F.E., Nocedal, J., Optimization methods for large-scale machine learning (2018) Siam Rev., 60 (2), pp. 223-311. , Jan; Kingma, D.P., Ba, J., (2014) Adam: A Method for Stochastic Optimization, , https://arxiv.org/abs/1412.6980","Alonso-Monsalve, S.; Department of Computer Science and Engineering, Spain; email: saul.alonso.monsalve@cern.ch",,,"Institute of Electrical and Electronics Engineers Inc.",,,,,2162237X,,,"32167911","English","IEEE Trans. Neural Networks Learn. Sys.",Article,"Final","All Open Access, Bronze, Green",Scopus,2-s2.0-85095340177
"Hong M., Li G., Zhang X., Huang Q.","57219971801;55714185400;57211151919;8435766200;","Generalized Zero-Shot Video Classification via Generative Adversarial Networks",2020,"MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia",,,,"2419","2426",,,"10.1145/3394171.3413517","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85096362047&doi=10.1145%2f3394171.3413517&partnerID=40&md5=79142bc6b6fd39ad1a4b16614b9c95c8","University of Chinese Academy of Sciences, School of Computer Science and Technology, Beijing, China; Key Lab of Big Data Mining and Knowledge Management, UCAS, Beijing, China; Key Lab of Intelligent Information Processing, ICT, CAS, Beijing, China","Hong, M., University of Chinese Academy of Sciences, School of Computer Science and Technology, Beijing, China; Li, G., University of Chinese Academy of Sciences, School of Computer Science and Technology, Beijing, China, Key Lab of Big Data Mining and Knowledge Management, UCAS, Beijing, China; Zhang, X., University of Chinese Academy of Sciences, School of Computer Science and Technology, Beijing, China, Key Lab of Big Data Mining and Knowledge Management, UCAS, Beijing, China; Huang, Q., University of Chinese Academy of Sciences, School of Computer Science and Technology, Beijing, China, Key Lab of Big Data Mining and Knowledge Management, UCAS, Beijing, China, Key Lab of Intelligent Information Processing, ICT, CAS, Beijing, China","Zero-shot learning (ZSL) is to classify images according to detailed attribute annotations into new categories that are unseen during the training stage. Generalized zero-shot learning (GZSL) adds seen categories to the test samples. Since the learned classifier has inherent bias against seen categories, GZSL is more challenging than traditional ZSL. However, at present, there is no detailed attribute description dataset for video classification. Therefore, the current zero-shot video classification problem is based on the synthesis of generative adversarial networks trained on seen-class features into unseen-class features for ZSL classification. In order to solve this problem, we propose a description text dataset based on the UCF101 action recognition dataset. To the best of our knowledge, this is the first work to add description of the classes to zero-shot video classification. We propose a new loss function that combines visual features with textual features. We extract text features from the proposed text data set, and constrain the process of generating synthetic features based on the principle that videos with similar text types should be similar. Our method reapplies the traditional zero-shot learning idea to video classification. From the experimental point of view, our proposed dataset and method have a positive impact on the generalized zero-shot video classification. © 2020 ACM.","computer vision; machine learning","Character recognition; Action recognition; Adversarial networks; Loss functions; Shot video classifications; Test samples; Textual features; Video classification; Visual feature; Classification (of information)",,,,,"QYZDJ-SSW-SYS013

National Natural Science Foundation of China, NSFC: 61620106009, 61772494, 61836002, 61931008, 61976069, U1636214

Youth Innovation Promotion Association of the Chinese Academy of Sciences","This work was supported in part by the Italy-China collaboration project TALENT:2018YFE0118400, in part by National Natural Science Foundation of China: 61620106009, 61772494, 61931008, U1636214, 61836002 and 61976069, in part by Key Research Program of Frontier Sciences, CAS: QYZDJ-SSW-SYS013, in part by Youth Innovation Promotion Association CAS.",,"Adler, J., Lunz, S., (2018) Banach Wasserstein GAN., pp. 6755-6764. , 2018; Akata, Z., Malinowski, M., Fritz, M., Schiele, B., (2016) Multicue Zero-Shot Learning with Strong Supervision., pp. 59-68. , 2016; Akata, Z., Perronnin, F., Harchaoui, Z., Schmid, C., (2013) Label-Embedding for Attribute-Based Classification., pp. 819-826. , 2013; Akata, Z., Perronnin, F., Harchaoui, Z., Schmid, C., Label-embedding for image classification (2016) IEEE Transactions on Pattern Analysis and Machine Intelligence, 38 (7), pp. 1425-1438. , 2016; Akata, Z., Reed, S., Walter, D.J, Lee, H., Schiele, B., (2015) Evaluation of Output Embeddings for Fine-grained Image Classification, pp. 2927-2936. , 2015; Arjovsky, M., Chintala, S., Bottou, L., Wasserstein gan (2017) CoRR, 2017. , http://arxiv.org/abs/1701.07875, 1701.07875; Brattoli, B., Buchler, U., Wahl, A., Schwab, M.E, Ommer, B., (2017) LSTM Self-Supervision for Detailed Behavior Analysis, pp. 3747-3756. , 2017; Carreira, J., Zisserman, A., (2017) Quo Vadis, Action Recognition-A New Model and the Kinetics Dataset, pp. 4724-4733. , 2017; Changpinyo, S., Chao, W., Gong, B., Sha, F., (2016) Synthesized Classifiers for Zero-Shot Learning, pp. 5327-5336. , 2016; Devries, T., Taylor, G.W, (2018) Learning Confidence for Out-of-Distribution Detection in Neural Networks, , 2018; Dong, J., Li, X., Xu, C., Ji, S., He, Y., Yang, G., Wang, X., (2019) Dual Encoding for Zero-Example Video Retrieval., pp. 9346-9355. , 2019; Elhoseiny, M., Elfeki, M., (2019) Creativity Inspired Zero-Shot Learning. ArXiv: Computer Vision and Pattern Recognition, , 2019; Elhoseiny, M., Saleh, B., Elgammal, A., (2013) Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions., pp. 2584-2591. , 2013; Elhoseiny, M., Zhu, Y., Zhang, H., Elgammal, A., (2017) Link the Head to the Beak: Zero Shot Learning from Noisy Text Description at Part Precision, 2017, pp. 6288-6297; Farhadi, A., Endres, I., Hoiem, D., Forsyth, D., (2009) Describing Objects by Their Attributes, pp. 1778-1785. , 2009; Felix, R., Vijay Kumar, B.G., Reid, I., Carneiro, G., (2018) Multimodal Cycle-Consistent Generalized Zero-Shot Learning, pp. 21-37. , 2018; Frome, A., Corrado, G.S, Shlens, J., Bengio, S., Dean, J., Ranzato, M., Mikolov, T., (2013) DeViSE: A Deep Visual-Semantic Embedding Model., pp. 2121-2129. , 2013; Fu, Y., Hospedales, T.M, Xiang, T., Fu, Z., Gong, S., (2014) Transductive Multi-view Embedding for Zero-Shot Recognition and Annotation., pp. 584-599. , 2014; Goodfellow, I., Pougetabadie, J., Mirza, M., Xu, B., Wardefarley, D., Ozair, S., Courville, A., Bengio, Y., (2014) Generative Adversarial Nets, pp. 2672-2680. , 2014; Guo, Y., Ding, G., Han, J., Gao, Y., (2017) Synthesizing Samples for Zero-shot Learning, pp. 1774-1780. , 2017; Guo, Y., Ding, G., Han, J., Gao, Y., Zero-shot learning with transferred samples (2017) IEEE Transactions on Image Processing, 26 (7), pp. 3277-3290. , 2017; Chandra Gupta, H., Dev Sharma, B., On non-additive measures of inaccuracy (1976) Czechoslovak Mathematical Journal, 26 (4), pp. 584-595. , 1976; Jurie, F., Bucher, M., Herbin, S., (2017) Generating Visual Representations for Zero-Shot Classification, pp. 2666-2673. , 2017; Kampffmeyer, M., Chen, Y., Liang, X., Wang, H., Zhang, Y., Xing, E.P, (2019) Rethinking Knowledge Graph Propagation for Zero-Shot Learning., pp. 11487-11496. , 2019; Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Feifei, L., (2014) Large-Scale Video Classification with Convolutional Neural Networks., pp. 1725-1732. , 2014; Kodirov, E., Xiang, T., Fu, Z., Gong, S., (2015) Unsupervised Domain Adaptation for Zero-Shot Learning., pp. 2452-2460. , 2015; Kodirov, E., Xiang, T., Gong, S., (2017) Semantic Autoencoder for Zero-Shot Learning, pp. 4447-4456. , 2017; Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T., (2011) HMDB: A Large Video Database for Human Motion Recognition., pp. 2556-2563. , 2011; Lampert, C.H, Nickisch, H., Harmeling, S., (2009) Learning to Detect Unseen Object Classes by Between-class Attribute Transfers, pp. 951-958; Lampert, C.H, Nickisch, H., Harmeling, S., Attribute-based classification for zero-shot visual object categorization (2014) IEEE Transactions on Pattern Analysis and Machine Intelligence, 36 (3), pp. 453-465. , 2014; Lee, C., Fang, W., Yeh, C., Frank Wang, Y., (2018) Multilabel Zero-Shot Learning with Structured Knowledge Graphs., pp. 1576-1585. , 2018; Lee, K., Lee, H., Lee, K., Shin, J., (2017) Training Confidencecalibrated Classifiers for Detecting Out-of-Distribution Samples, , Machine Learning, 2017; Liu, J., Kuipers, B., Savarese, S., (2011) Recognizing Human Actions by Attributes, pp. 3337-3344. , 2011; Long, Y., Liu, L., Shao, L., Shen, F., Ding, G., Han, J., (2017) From Zero-Shot Learning to Conventional Supervised Classification: Unseen Visual Data Synthesis., pp. 6165-6174. , 2017; Long, Y., Liu, L., Shen, F., Shao, L., Li, X., Zero-shot learning using synthesised unseen visual data with diffusion regularisation (2018) IEEE Transactions on Pattern Analysis and Machine Intelligence, 40 (10), pp. 2498-2512. , 2018; Mandal, D., Narayan, S., Kumar Dwivedi, S., Gupta, V., Ahmed, S., Shahbaz Khan, F., Shao, L., (2019) Out-Of-Distribution Detection for Generalized Zero-Shot Action Recognition, pp. 9985-9993. , 2019; Mirza, M., Osindero, S., (2014) Conditional Generative Adversarial Nets, , 2014; Kumar Mishra, A., Kumar Verma, V., Shiva Krishna Reddy, M., Arulkumar, S., Rai, P., Mittal, A., (2018) A Generative Approach to Zero-Shot and Few-Shot Action Recognition, pp. 372-380. , 2018; Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome, A., Corrado, G.S, Dean, J., (2013) Zero-Shot Learning by Convex Combination of Semantic Embeddings, , Learning, 2013; Pan, J., Wang, C., Jia, X., Shao, J., Sheng, L., Yan, J., Wang, X., (2019) Video Generation from Single Semantic Label Map, , Computer Vision and Pattern Recognition, 2019; Qin, J., Liu, L., Shao, L., Shen, F., Ni, B., Chen, J., Wang, Y., (2017) Zero-Shot Action Recognition with Error-Correcting Output Codes, pp. 1042-1051. , 2017; Salton, G., Buckley, C., Term-weighting approaches in automatic text retrieval (1988) Information Processing and Management, 24 (5), pp. 323-328; Soomro, K., Roshan Zamir, A., Shah, M., (2012) UCF101: A Dataset of 101 Human Actions Classes from Videos in the Wild, , Computer Vision and Pattern Recognition, 2012; Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M., (2015) Learning Spatiotemporal Features with 3D Convolutional Networks., pp. 4489-4497. , 2015; Wang, X., Ye, Y., Gupta, A., (2018) Zero-Shot Recognition Via Semantic Embeddings and Knowledge Graphs., pp. 6857-6866. , 2018; Xian, Y., Lorenz, T., Schiele, B., Akata, Z., (2018) Feature Generating Networks for Zero-Shot Learning, pp. 5542-5551. , 2018; Xian, Y., Schiele, B., Akata, Z., (2017) Zero-Shot Learning-the Good, the Bad and the Ugly, pp. 3077-3086. , 2017; Xu, X., Hospedales, T.M, Gong, S., Transductive zero-shot action recognition by word-vector embedding (2017) International Journal of Computer Vision 123, 3, pp. 309-333. , 2017; Zhang, B., Li, L., Yang, S., Wang, S., Zha, Z., Huang, Q., State-relabeling adversarial active learning (2020) Proceedings of the IEEE International Conference on Computer Vision; Zhang, C., Peng, Y., (2018) Visual Data Synthesis Via GAN for Zero-Shot Video Classification., pp. 1128-1134. , 2018; Zhu, J., Park, T., Isola, P., Efros, A.A, (2017) Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks, pp. 2242-2251. , 2017","Li, G.; University of Chinese Academy of Sciences, China; email: liguorong@ucas.ac.cn
Li, G.; Key Lab of Big Data Mining and Knowledge Management, China; email: liguorong@ucas.ac.cn",,"ACM SIGMM","Association for Computing Machinery, Inc","28th ACM International Conference on Multimedia, MM 2020","12 October 2020 through 16 October 2020",,163870,,9781450379885,,,"English","MM - Proc. ACM Int. Conf. Multimed.",Conference Paper,"Final","",Scopus,2-s2.0-85096362047
"van Steenkiste S., Kurach K., Schmidhuber J., Gelly S.","57191041904;55566223600;7003514621;55038126700;","Investigating object compositionality in Generative Adversarial Networks",2020,"Neural Networks","130",,,"309","325",,3,"10.1016/j.neunet.2020.07.007","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85088638451&doi=10.1016%2fj.neunet.2020.07.007&partnerID=40&md5=9242e5ad97db95da45d85f9f157019be","IDSIA, SUPSI & USI, Via Cantonale 2C, Manno, 6928, Switzerland; Google Brain, Brandschenkestrasse 110, Zürich, 8002, Switzerland","van Steenkiste, S., IDSIA, SUPSI & USI, Via Cantonale 2C, Manno, 6928, Switzerland; Kurach, K., Google Brain, Brandschenkestrasse 110, Zürich, 8002, Switzerland; Schmidhuber, J., IDSIA, SUPSI & USI, Via Cantonale 2C, Manno, 6928, Switzerland; Gelly, S., Google Brain, Brandschenkestrasse 110, Zürich, 8002, Switzerland","Deep generative models seek to recover the process with which the observed data was generated. They may be used to synthesize new samples or to subsequently extract representations. Successful approaches in the domain of images are driven by several core inductive biases. However, a bias to account for the compositional way in which humans structure a visual scene in terms of objects has frequently been overlooked. In this work, we investigate object compositionality as an inductive bias for Generative Adversarial Networks (GANs). We present a minimal modification of a standard generator to incorporate this inductive bias and find that it reliably learns to generate images as compositions of objects. Using this general design as a backbone, we then propose two useful extensions to incorporate dependencies among objects and background. We extensively evaluate our approach on several multi-object image datasets and highlight the merits of incorporating structure for representation learning purposes. In particular, we find that our structured GANs are better at generating multi-object images that are more faithful to the reference distribution. More so, we demonstrate how, by leveraging the structure of the learned generative process, one can ‘invert’ the learned generative model to perform unsupervised instance segmentation. On the challenging CLEVR dataset, it is shown how our approach is able to improve over other recent purely unsupervised object-centric approaches to image generation. © 2020 Elsevier Ltd","Compositionality; Generative Adversarial Networks; Generative modeling; Instance segmentation; Objects; Representation learning","Artificial intelligence; Cognitive systems; Adversarial networks; Compositionality; General designs; Generative model; Generative process; Image datasets; Image generations; Inductive bias; Image enhancement; article; feature learning (machine learning); human; automated pattern recognition; image processing; procedures; Humans; Image Processing, Computer-Assisted; Neural Networks, Computer; Pattern Recognition, Automated",,,,,"International Business Machines Corporation, IBM

Nvidia

Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung, SNF: 200021_165675/1","The authors wish to thank Damien Vincent, Alexander Kolesnikov, Olivier Bachem, Klaus Greff, and Paulo Rauber for helpful comments and constructive feedback. The authors are grateful to Marcin Michalski and Pierre Ruyssen for their technical support. This research was in part supported by the Swiss National Science Foundation grant 200021_165675/1 , and by hardware donations from NVIDIA Corporation as part of the Pioneers of AI Research award, and by IBM.",,"Arandjelović, R., Zisserman, A., Object discovery with a copy-pasting GAN (2019), arXiv preprint; Arjovsky, M., Chintala, S., Bottou, L., (2017), Wasserstein generative adversarial networks. In International conference on machine learning (pp. 214–223); Azadi, S., Pathak, D., Ebrahimi, S., Darrell, T., Compositional GAN: Learning image-conditional binary composition (2019), arXiv preprint; Ba, J.L., Kiros, J.R., Hinton, G.E., Layer normalization (2016), arXiv preprint; Battaglia, P.W., Hamrick, J.B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., Relational inductive biases, deep learning, and graph networks (2018), arXiv preprint; Battaglia, P.W., Hamrick, J.B., Tenenbaum, J.B., Simulation as an engine of physical scene understanding (2013) Proceedings of the National Academy of Sciences; Bengio, Y., Courville, A., Vincent, P., Representation learning: A review and new perspectives (2013) IEEE Transactions on Pattern Analysis and Machine Intelligence, 35 (8), pp. 1798-1828; Bielski, A., Favaro, P., Emergence of object segmentation in perturbed generative models (2019) Advances in neural information processing systems 32, pp. 7254-7264; Chen, M., Artières, T., Denoyer, L., Unsupervised object segmentation by redrawing (2019) Advances in neural information processing systems 32, pp. 12705-12716; Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P., Infogan: Interpretable representation learning by information maximizing generative adversarial nets (2016) Advances in neural information processing systems, pp. 2172-2180; Dinh, L., Sohl-Dickstein, J., Bengio, S., (2017), Density estimation using Real NVP. In Fifth international conference on learning representations; Donahue, J., Krähenbühl, P., Darrell, T., (2017), Adversarial feature learning. In Fifth international conference on learning representations; Dumoulin, V., Belghazi, I., Poole, B., Mastropietro, O., Lamb, A., Arjovsky, M., (2017), Adversarially learned inference. In Fifth international conference on learning representations; Eslami, S.A., Heess, N., Weber, T., Tassa, Y., Szepesvari, D., Hinton, G.E., Attend, infer, repeat: Fast scene understanding with generative models (2016) Advances in neural information processing systems, pp. 3225-3233; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Generative adversarial nets (2014) Advances in neural information processing systems, pp. 2672-2680; Greff, K., Kaufman, R.L., Kabra, R., Watters, N., Burgess, C., Zoran, D., (2019), Multi-Object representation learning with iterative variational inference. In International conference on machine learning (pp. 2424–2433); Greff, K., Rasmus, A., Berglund, M., Hao, T., Valpola, H., Schmidhuber, J., Tagger: Deep unsupervised perceptual grouping (2016) Advances in neural information processing systems, pp. 4484-4492; Greff, K., van Steenkiste, S., Schmidhuber, J., Neural expectation maximization (2017) Advances in neural information processing systems, pp. 6691-6701; Gregor, K., Danihelka, I., Graves, A., Rezende, D., Wierstra, D.D., (2015), A recurrent neural network for image generation. In International conference on machine learning (pp. 1462–1471); Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C., Improved training of Wasserstein GANs (2017) Advances in neural information processing systems, pp. 5767-5777; Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S., GANs trained by a two time-scale update rule converge to a local Nash equilibrium (2017) Advances in neural information processing systems, pp. 6626-6637; Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., (2017), beta-VAE: Learning basic visual concepts with a constrained variational framework. In Fifth international conference on learning representations; Hinz, T., Heinrich, S., Wermter, S., (2019), Generating multiple objects at spatially distinct locations. In International conference on learning representations; Hubert, L., Arabie, P., Comparing partitions (1985) Journal of Classification, 2 (1), pp. 193-218; Im, D.J., Kim, C.D., Jiang, H., Memisevic, R., Generating images with recurrent adversarial networks (2016), arXiv preprint; Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A., Image-to-image translation with conditional adversarial networks (2017) 2017 IEEE conference on computer vision and pattern recognition, pp. 5967-5976. , IEEE; Janner, M., Levine, S., Freeman, W.T., Tenenbaum, J.B., Finn, C., Wu, J., (2019), Reasoning about physical interactions with object-centric models. In International conference on learning representations; Johnson, J., Gupta, A., (2018), Fei-Fei, L. Image generation from scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1219–1228); Johnson, J., Hariharan, B., (2017), van der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., & Girshick, R. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2901–2910); Kingma, D.P., Ba, J., (2015), Adam: A method for stochastic optimization. In Third international conference on learning representations; Kingma, D.P., Welling, M., (2014), Stochastic gradient VB and the variational auto-encoder. In Second international conference on learning representations; Kosiorek, A., Kim, H., Teh, Y.W., Posner, I., Sequential attend, infer, repeat: Generative modelling of moving objects (2018) Advances in neural information processing systems, pp. 8606-8616; Krizhevsky, A., Hinton, G., Learning multiple layers of features from tiny images: Technical Report (2009), Citeseer; Kurach, K., Lučić, M., Zhai, X., Michalski, M., Gelly, S., (2019), A large-scale study on regularization and normalization in GANs. In International conference on machine learning (pp. 3581–3590); Kwak, H., Zhang, B.-T., Generating images part by part with composite generative adversarial networks (2016), arXiv preprint; Le Roux, N., Heess, N., Shotton, J., Winn, J., Learning a generative model of images by factoring appearance and shape (2011) Neural Computation, 23 (3), pp. 593-650; LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., Gradient-based learning applied to document recognition (1998) Proceedings of the IEEE, 86 (11), pp. 2278-2324; Lin, C.-H., Yumer, E., Wang, O., Shechtman, E., Lucey, S.S.-G., (2018), Spatial transformer generative adversarial networks for image compositing. In Proceedings of the ieee conference on computer vision and pattern recognition (pp. 9455–9464); Lucic, M., Kurach, K., Michalski, M., Gelly, S., Bousquet, O., Are gans created equal? A large-scale study (2018) Advances in neural information processing systems, pp. 700-709; Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y., (2018), Spectral normalization for generative adversarial networks. In International conference on learning representations; Nash, C., Eslami, S.A., Burgess, C., Higgins, I., Zoran, D., Weber, T., (2017), The multi-entity variational autoencoder. In NIPS workshop on learning disentangled representations: from perception to control; Radford, A., Metz, L., Chintala, S., Unsupervised representation learning with deep convolutional generative adversarial networks (2015), arXiv preprint; Santoro, A., Raposo, D., Barrett, D.G., Malinowski, M., Pascanu, R., Battaglia, P., A simple neural network module for relational reasoning (2017) Advances in neural information processing systems, pp. 4967-4976; Schmidhuber, J., Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments: Technical Report FKI-126 90 (1990), Institut für Informatik, Technische Universität München; Schmidhuber, J., Learning factorial codes by predictability minimization (1992) Neural Computation, 4 (6), pp. 863-879; Schmidhuber, J., Generative adversarial networks are special cases of artificial curiosity (1990) and also closely related to predictability minimization (1991) (2020) Neural Networks; Spampinato, C., Palazzo, S., D'Oro, P., Giordano, D., Shah, M., Adversarial framework for unsupervised learning of motion dynamics in videos (2019) International Journal of Computer Vision, pp. 1-20; Spelke, E.S., Kinzler, K.D., Core knowledge (2007) Developmental Science, 10 (1), pp. 89-96; (2018), van Steenkiste, S., Chang, M., Greff, K., & Schmidhuber, J. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In International conference on learning representations; van Steenkiste, S., Kurach, K., & Gelly, S. (2018b). A case for object compositionality in deep generative models of images. In NeurIPS workshop on modeling the physical world: Learning, perception, and control; (2016), Van Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. Pixel recurrent neural networks. In International conference on machine learning (pp. 1747–1756); Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Attention is all you need (2017) Advances in neural information processing systems, pp. 5998-6008; Xu, K., Liang, H., Zhu, J., Su, H., Zhang, B., Deep structured generative models (2018), arXiv preprint; Yang, J., Kannan, A., Batra, D., Parikh, D.L.-G., (2017), Layered recursive generative adversarial networks for image generation. In Fifth international conference on learning representations; Zambaldi, V., Raposo, D., Santoro, A., Bapst, V., Li, Y., Babuschkin, I., (2019), Deep reinforcement learning with relational inductive biases. In International conference on learning representations","van Steenkiste, S.; IDSIA, Via Cantonale 2C, Switzerland; email: sjoerd@idsia.ch",,,"Elsevier Ltd",,,,,08936080,,NNETE,"32736226","English","Neural Netw.",Article,"Final","All Open Access, Green",Scopus,2-s2.0-85088638451
"Chen J., Mowlaei M.E., Shi X.","56556575800;57063386900;55228140300;","Population-scale Genomic Data Augmentation Based on Conditional Generative Adversarial Networks",2020,"Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020",,,,"","",,,"10.1145/3388440.3412475","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85096997585&doi=10.1145%2f3388440.3412475&partnerID=40&md5=3c954c0ea6aa79769f73e1fb839163ac","Temple University, Philadelphia, PA, United States","Chen, J., Temple University, Philadelphia, PA, United States; Mowlaei, M.E., Temple University, Philadelphia, PA, United States; Shi, X., Temple University, Philadelphia, PA, United States","Although next generation sequencing technologies have made it possible to quickly generate a large collection of sequences, current genomic data still suffer from small data sizes, imbalances, and biases due to various factors including disease rareness, test affordability, and concerns about privacy and security. In order to address these limitations of genomic data, we develop a Population-scale Genomic Data Augmentation based on Conditional Generative Adversarial Networks (PG-cGAN) to enhance the amount and diversity of genomic data by transforming samples already in the data rather than collecting new samples. Both the generator and discriminator in the PG-CGAN are stacked with convolutional layers to capture the underlying population structure. Our results for augmenting genotypes in human leukocyte antigen (HLA) regions showed that PC-cGAN can generate new genotypes with similar population structure, variant frequency distributions and LD patterns. Since the input for PC-cGAN is the original genomic data without assumptions about prior knowledge, it can be extended to enrich many other types of biomedical data and beyond. © 2020 ACM.","data augmentation; deep learning; generative adversarial networks; genomics; machine learning","Bioinformatics; Genes; Medical informatics; Metadata; Adversarial networks; Biomedical data; Frequency distributions; Human leukocyte antigen; Next-generation sequencing; Population structures; Prior knowledge; Privacy and security; Population statistics",,,,,"National Science Foundation, NSF: 1750632","This work is partially supported by the National Science Foundation of the United States (Award Number: 1750632).",,"Abadi, M., Chu, A., Goodfellow, I., Brendan McMahan, H., Mironov, I., Talwar, K., Zhang, L., 2016. Deep learning with differential privacy Proceedings of the 2016 Acmsigsac Conference on Computer and Communications Security, pp. 308-318; Alyafi, B., Diaz, O., Martí, R., 2020. DCGANs for realistic breast mass augmentation in x-ray mammography Medical Imaging 2020: Computer-Aided Diagnosis, 11314, p. 1131420. , International Society for Optics and Photonics; Antoniou, A., Storkey, A., Edwards, H., (2017) Data Augmentation Generative Adversarial Networks, 2017. , arXiv preprint arXiv 1711.04340; Arjovsky, M., Chintala, S., Bottou, L., (2017) Wasserstein Gan, 2017. , arXiv preprint arXiv 1701.07875; Bailo, O., Shik Ham, D., Min Shin, Y., 2019. Red blood cell image generation for data augmentation using conditional generative adversarial networks Proceedings of the Ieee Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0; Beaulieu-Jones, B.K., Steven Wu, Z., Williams, C., Lee, R., Bhavnani, S.P., Brian Byrd, J., Greene, C.S., 2019 Privacy-preserving generative deep neural networks support clinical data sharing (2019) Circulation: Cardiovascular Quality and Outcomes, 12 (7), p. e005122; Berthelot, D., Schumm, T., Metz, L., (2017) Began: Boundary Equilibrium Generative Adversarial Networks, 2017. , arXiv preprint arXiv 1703.10717; Chaudhari, P., Agrawal, H., Kotecha, K., Data augmentation using MG-GAN for improved cancer classification on gene expression data (2019) Soft Computing, 2019, pp. 1-11; Chen, J., Shi, X., Sparse Convolutional Denoising Autoencoders for Genotype Imputation (2019) Genes, 10 (9-2019), p. 652; Chen, J., Shi, X., 2019. A Sparse Convolutional Predictor with Denoising Autoencoders for Phenotype Prediction Proceedings of the 10th Acm International Conference on Bioinformatics, Computational Biology and Health Informatics, pp. 217-222; Chen, Y., Li, Y., Narayan, R., Subramanian, A., Xie, X., Gene expression inference with deep learning (2016) Bioinformatics, 32 (12-2016), pp. 1832-1839; A global reference for human genetic variation (2015) Nature, 526 (7571), pp. 68-74. , 1000 Genomes Project Consortium et al. 2015; Frid-Adar, M., Klang, E., Amitai, M., Goldberger, J., Greenspan, H., (2018) Gan-based Data Augmentation for Improved Liver Lesion Classification, 2018; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., 2014. Generative adversarial nets Advances in Neural Information Processing Systems, pp. 2672-2680; Haller, B.C., Messer, P.W., 2019 SLiM 3: Forward genetic simulations beyond the Wright-Fisher model (2019) Molecular Biology and Evolution, 36 (3), pp. 632-637; Han, C., Hayashi, H., Rundo, L., Araki, R., Shimoda, W., Muramatsu, S., Furukawa, Y., Nakayama, H., 2018. GAN-based synthetic brain MR image generation 2018 Ieee 15th International Symposium on Biomedical Imaging (ISBI 2018). Ieee, pp. 734-738; Hernandez, R.D., Uricchio, L.H., Hartman, K., Ye, C., Dahl, A., Zaitlen, N., Ultra-rare variants drive substantial cis-heritability of human gene expression (2019) BioRxiv, 2019, p. 219238; Hillert, J., 1994. Human leukocyte antigen studies in multiple sclerosis. Annals of Neurology (1994) Official Journal of the American Neurological Association and the Child Neurology Society, 36 (1), pp. S15-S17; Karras, T., Aila, T., Laine, S., Lehtinen, J., (2017) Progressive Growing of Gans for Improved Quality, Stability, and Variation, 2017. , arXiv preprint arXiv 1710.10196; Kelleher, J., Etheridge, A.M., McVean, G., Efficient coalescent simulation and genealogical analysis for large sample sizes (2016) PLoS Computational Biology, 12 (5), p. 2016; Koepfli, K., Paten, B., Genome 10K Community of Scientists, and Stephen J O?Brien (2015) 2015. The Genome 10K Project: A Way Forward. Annu. Rev. Anim. Biosci, 3 (1), pp. 57-111; Li, X., Chen, L., Wang, L., Wu, P., Tong, W., 2018. SCGAN: Disentangled Representation Learning by Adding Similarity Constraint on Generative Adversarial Nets Ieee Access, 7 (2018), pp. 147928-147938; Lunshof, J.E., Chadwick, R., Vorhaus, D.B., Church, G.M., From genetic privacy to open consent (2008) Nature Reviews Genetics, 9 (5), pp. 406-411; Marouf, M., MacHart, P., Bansal, V., Kilian, C., Magruder, D.S., Krebs, C.F., Bonn, S., Realistic in silico generation and augmentation of single-cell RNA-seq data using generative adversarial networks (2020) Nature Communications, 11 (1-2020), pp. 1-12; Min, S., Lee, B., Yoon, S., Deep learning in bioinformatics (2017) Briefings in Bioinformatics, 18 (5-2017), pp. 851-869; Mirza, M., Osindero, S., (2014) Conditional Generative Adversarial Nets, 2014. , arXiv preprint arXiv 1411.1784; Nordborg, M., Tavare, S., Linkage disequilibrium: What history has to tell us (2002) Trends in Genetics, 18 (2-2002), pp. 83-90; Novembre, J., Stephens, M., Interpreting principal component analyses of spatial population genetic variation (2008) Nature Genetics, 40 (5-2008), pp. 646-649; Patterson, N., Price, A.L., Reich, D., Population structure and eigenanalysis (2006) PLoS Genetics, 2 (12), p. 2006; Price, A.L., Zaitlen, N.A., Reich, D., Patterson, N., New approaches to population stratification in genome-wide association studies (2010) Nature Reviews Genetics, 11 (7-2010), pp. 459-463; Radford, A., Metz, L., Chintala, S., (2015) Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, 2015. , arXiv preprint arXiv 1511.06434; Reich, D., Price, A.L., Patterson, N., Principal component analysis of genetic data (2008) Nature Genetics, 40 (5-2008), pp. 491-492; Reich, D.E., Cargill, M., Bolk, S., Ireland, J., Sabeti, P.C., Richter, D.J., Lavery, T., Ward, R., Linkage disequilibrium in the human genome (2001) Nature, 411 (6834), pp. 199-204; Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Landray, M., UK biobank: An open access resource for identifying the causes of a wide range of complex diseases of middle and old age (2015) PLoS Medicine, 12 (3), p. 2015; Van Dyk, D.A., Meng, X., 2001. The art of data augmentation (2001) Journal of Computational and Graphical Statistics, 10 (1), pp. 1-50; Zhu, J., Park, T., Isola, P., Efros, A.A., 2017. Unpaired image-To-image translation using cycle-consistent adversarial networks Proceedings of the Ieee International Conference on Computer Vision, pp. 2223-2232; Zhu, X., Liu, Y., Li, J., Wan, T., Qin, Z., 2018. Emotion classification with data augmentation using generative adversarial networks Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 349-360. , Springer",,,"ACM Special Interest Group on Bioinformatics (SIGBio)","Association for Computing Machinery, Inc","11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB 2020","21 September 2020 through 24 September 2020",,164833,,9781450379649,,,"English","Proc. ACM Int. Conf. Bioinformatics, Computational Biology Health Informatics, BCB",Conference Paper,"Final","",Scopus,2-s2.0-85096997585
"Tschuchnig M.E., Oostingh G.J., Gadermayr M.","57211430456;6507748784;50361333000;","Generative Adversarial Networks in Digital Pathology: A Survey on Trends and Future Potential",2020,"Patterns","1","6","100089","","",,7,"10.1016/j.patter.2020.100089","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85097024499&doi=10.1016%2fj.patter.2020.100089&partnerID=40&md5=dcf17623d4ff2a4084634dde4271e52d","Department of Information Technologies and Systems Management, Salzburg University of Applied Sciences, Puch bei Hallein, 5412, Austria; Department of Biomedical Sciences, Salzburg University of Applied Sciences, Puch bei Hallein, 5412, Austria","Tschuchnig, M.E., Department of Information Technologies and Systems Management, Salzburg University of Applied Sciences, Puch bei Hallein, 5412, Austria, Department of Biomedical Sciences, Salzburg University of Applied Sciences, Puch bei Hallein, 5412, Austria; Oostingh, G.J., Department of Biomedical Sciences, Salzburg University of Applied Sciences, Puch bei Hallein, 5412, Austria; Gadermayr, M., Department of Information Technologies and Systems Management, Salzburg University of Applied Sciences, Puch bei Hallein, 5412, Austria","Image analysis in the field of digital pathology has recently gained increased popularity. The use of high-quality whole-slide scanners enables the fast acquisition of large amounts of image data, showing extensive context and microscopic detail at the same time. Simultaneously, novel machine-learning algorithms have boosted the performance of image analysis approaches. In this paper, we focus on a particularly powerful class of architectures, the so-called generative adversarial networks (GANs) applied to histological image data. Besides improving performance, GANs also enable previously intractable application scenarios in this field. However, GANs could exhibit a potential for introducing bias. Hereby, we summarize the recent state-of-the-art developments in a generalizing notation, present the main applications of GANs, and give an outlook of some chosen promising approaches and their possible future applications. In addition, we identify currently unavailable methods with potential for future applications. The use of high-quality whole-slide scanners enables the fast acquisition of large amounts of image data, showing extensive context and microscopic detail at the same time. While manual examination of these images of considerable size is highly time consuming and error prone, state-of-the-art machine-learning approaches enable efficient, automated processing of whole-slide images. In this paper, we focus on a particularly powerful class of deep-learning architectures, the so-called generative adversarial networks. Over the past years, the high number of publications on this topic indicates a very high potential of generative adversarial networks in the field of digital pathology. In this survey, the most important publications are collected and categorized according to the techniques used and the aspired application scenario. We identify the main ideas and provide an outlook into the future. Whole-slide scanners digitize microscopic tissue slides and thereby generate a large amount of digital image material. This advocates for methods facilitating (semi-)automated analysis. In this paper, we investigate generative adversarial networks, which are a powerful class of deep-learning-based approaches, useful in, for example, histological image analysis. The most important publications in the field of digital pathology are collected, summarized, and categorized according to the technical approaches employed and the aspired application scenarios. We identify the main findings and furthermore provide an outlook. © 2020 The Authors","computational pathology; DSML 1: Concept: Basic principles of a new data science output observed and reported; generative adversarial network; histology; image-to-image translation; survey","Deep learning; Learning algorithms; Learning systems; Network architecture; Pathology; Scanning; Surveys; Adversarial networks; Application scenario; Automated processing; Digital pathologies; Improving performance; Learning architectures; Learning-based approach; Machine learning approaches; Image analysis",,,,,"FHS-2019-10-KIAMed","This work was partially funded by the County of Salzburg under grant number FHS-2019-10-KIAMed .",,"Kooi, T., Litjens, G., van Ginneken, B., Gubern-Mérida, A., Sánchez, C.I., Mann, R., den Heeten, A., Karssemeijer, N., Large scale deep learning for computer aided detection of mammographic lesions (2017) Med. Image Anal., 35, pp. 303-312; Tsuda, H., Akiyama, F., Kurosumi, M., Sakamoto, G., Yamashiro, K., Oyama, T., Hasebe, T., Umemura, S., Evaluation of the interobserver agreement in the number of mitotic figures breast carcinoma as simulation of quality monitoring in the Japan national surgical adjuvant study of breast cancer (NSAS-BC) protocol (2000) Jpn. J. Cancer Res., 91, pp. 451-457; Persson, J., Wilderäng, U., Jiborn, T., Wiklund, P., Damber, J., Hugosson, J., Steineck, G., Bjartell, A., Interobserver variability in the pathological assessment of radical prostatectomy specimens: findings of the laparoscopic prostatectomy robot open (LAPPRO) study (2014) Scand. J. Urol., 48, pp. 160-167; Metter, D.M., Colgan, T.J., Leung, S.T., Timmons, C.F., Park, J.Y., Trends in the US and Canadian pathologist workforces from 2007 to 2017 (2019) JAMA Netw. Open, 2, p. e194337; Petriceks, A.H., Salmi, D., Trends in pathology graduate medical education programs and positions, 2001 to 2017 (2018) Acad. Pathol., 5; Mahmood, F., Borders, D., Chen, R., McKay, G.N., Salimian, K.J., Baras, A., Durr, N.J., Deep adversarial training for multi-organ nuclei segmentation in histopathology images (2019) IEEE Trans. Med. Imag., p. 1; Jung, C., Kim, C., Chae, S.W., Oh, S., Unsupervised segmentation of overlapped nuclei using Bayesian classification (2010) IEEE Trans. Biomed. Eng., 57, pp. 2825-2832; Bug, D., Gräbel, P., Feuerhake, F., Oswald, E., Schüler, J., Merhof, D., (2019), Supervised and unsupervised cell-nuclei detection in immunohistology, in Proceedings of the 2nd MICCAI Workshop on Computational Pathology (COMPAY); Xu, Y., Zhu, J.-Y., Chang, E.I.-C., Lai, M., Tu, Z., Weakly supervised histopathology cancer image segmentation and classification (2014) Med. Image Anal., 18, pp. 591-604; Jiang, J., Hu, Y.-C., Tyagi, N., Zhang, P., Rimner, A., Mageras, G.S., Deasy, J.O., Veeraraghavan, H., (2018), pp. 777-785. , Tumor-aware, adversarial domain adaptation from CT to MRI for lung cancer segmentation, in Proceedings of the Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI’18); Sirinukunwattana, K., Pluim, J.P., Chen, H., Qi, X., Heng, P.-A., Guo, Y.B., Wang, L.Y., Sanchez, U., Gland segmentation in colon histology images: the GlaS challenge contest (2017) Med. Image Anal., 35, pp. 489-502; BenTaieb, A., Hamarneh, G., (2016), pp. 460-468. , Topology aware fully convolutional networks for histology gland segmentation, in Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI’16); Hou, L., Samaras, D., Kurc, T.M., Gao, Y., Davis, J.E., Saltz, J.H., (2016), Patch-based convolutional neural network for whole-slide tissue image classification, in Proceedings of the International Conference on Computer Vision (CVPR’16); Gecer, B., Aksoy, S., Mercan, E., Shapiro, L.G., Weaver, D.L., Elmore, J.G., Detection and classification of cancer in whole slide breast histopathology images using deep convolutional networks (2018) Pattern Recognition, 84, pp. 345-356; Khan, A.M., Rajpoot, N., Treanor, D., Magee, D., A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution (2014) IEEE Trans. Biomed. Eng., 61, pp. 1729-1738; Macenko, M., Niethammer, M., Marron, J.S., Borland, D., Woosley, J.T., Guan, X., Schmitt, C., Thomas, N.E., (2009), pp. 1107-1110. , A method for normalizing histology slides for quantitative analysis, in Proceedings of the IEEE International Symposium on Biomedical Imaging: From Nano to Macro (ISBI’09); Reinhard, E., Ashikhmin, M., Gooch, B., Shirley, P., Color transfer between images (2001) IEEE Comput. Graph. Appl., 21, pp. 34-41; Tellez, D., Litjens, G., Bándi, P., Bulten, W., Bokhorst, J.-M., Ciompi, F., van der Laak, J., Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology (2019) Med. Image Anal., 58, p. 101544; Kowal, M., Żejmo, M., Skobel, M., Korbicz, J., Monczak, R., Cell nuclei segmentation in cytological images using convolutional neural network and seeded watershed algorithm (2019) J. Digital Imag., 33, pp. 231-242; Abdolhoseini, M., Kluge, M.G., Walker, F.R., Johnson, S.J., Segmentation of heavily clustered nuclei from histopathological images (2019) Sci. Rep., 9; Mosaliganti, K., Cooper, L., Sharp, R., Machiraju, R., Leone, G., Huang, K., Saltz, J., Reconstruction of cellular biological structures from optical microscopy data (2008) IEEE Trans. Vis. Comput. Graph., 14, pp. 863-876; Ojala, T., Pietikäinen, M., Mäenpää, T., Multiresolution gray-scale and rotation invariant texture classification with local binary patterns (2002) IEEE Trans. Pattern Anal. Mach. Intell., 24, pp. 971-987; Sánchez, J., Perronnin, F., Mensink, T., Verbeek, J.J., Image classification with the Fisher vector: theory and practice (2013) Int. J. Comput. Vis., 105, pp. 222-245; Gadermayr, M., Strauch, M., Klinkhammer, B., Djudjaj, S., Boor, P., Merhof, D., (2016), pp. 616-622. , Domain adaptive classification for compensating variability in histopathological whole slide images, in Proceedings of the International Conference on Image Analysis and Recognition (ICIAR’16); Dimitriou, N., Arandjelović, O., Caie, P.D., Deep learning for whole slide image analysis: an overview (2019) Front. Med., 6; Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., van der Laak, J.A., Sánchez, C.I., A survey on deep learning in medical image analysis (2017) Med. Image Anal., 42, pp. 60-88; Ronneberger, O., Fischer, P., Brox, T., (2015), pp. 234-241. , U-net: Convolutional networks for biomedical image segmentation, in Proceedings of the International Conference on Medical Image Computing and Computer Aided Interventions (MICCAI’15); Gadermayr, M., Dombrowski, A.-K., Klinkhammer, B.M., Boor, P., Merhof, D., CNN cascades for segmenting sparse objects in gigapixel whole slide images (2019) Comput. Med. Imaging Graphics, 71, pp. 40-48; Wei, J., Suriawinata, A., Vaickus, L., Ren, B., Liu, X., Wei, J., Hassanpour, S., (2019), Generative image translation for data augmentation in colorectal histopathology images, in Proceedings of the NeurIPS workshop on Machine Learning for Health; Zhao, A., Balakrishnan, G., Durand, F., Guttag, J.V., Dalca, A.V., (2019), Data augmentation using learned transforms for one-shot medical image segmentation, in Proceedings of the International Conference on Computer Vision (CVPR); Gadermayr, M., Gupta, L., Appel, V., Boor, P., Klinkhammer, B.M., Merhof, D., Generative adversarial networks for facilitating stain-independent supervised and unsupervised segmentation: a study on kidney histology (2019) IEEE Trans. Med. Imaging, 38, pp. 2293-2302; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Advances in Neural Information Processing Systems, pp. 2672-2680. , Z. Gharamani M. Welling C. Cortes N.D. Lawrence K.Q. Weinberger NIPS; Hou, L., Agarwal, A., Samaras, D., Kurc, T.M., Gupta, R.R., Saltz, J.H., (2019), Robust histopathology image analysis: To label or to synthesize?, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19); Ren, J., Hacihaliloglu, I., Singer, E.A., Foran, D.J., Qi, X., “Adversarial domain adaptation for classification of prostatehistopathology whole-slide images,” in Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (2018), pp. 201-209; Zanjani, F.G., Zinger, S., Bejnordi, B.E., (2018), J.A.W.M. van der Laak, and P.H.N. de With, Stain normalization of histopathology images using generative adversarial networks, in Proceedings of the 15th International Symposium on Biomedical Imaging (ISBI’18), IEEE; Shaban, M.T., Baur, C., Navab, N., Albarqouni, S., (2019), Staingan: Stain style transfer for digital histological images, in Proceedings of the 16th International Symposium on Biomedical Imaging (ISBI’19), IEEE; (2019), 102, pp. 151-163. , T. de Bel, M. Hermsen, J. Kers, J. van der Laak, and G. Litjens, Stain-transforming cycle-consistent generative adversarial networks for improved segmentation of renal histopathology, in Proceedings of the 2nd International Conference on Medical Imaging with Deep Learning (MIDL’19), PMLR; Quiros, A.C., Murray-Smith, R., Yuan, K., Pathology GAN: learning deep representations of cancer tissue (2019) arXiv, , 1907.02644; Huo, Y., Xu, Z., Bao, S., Assad, A., Abramson, R.G., Landman, B.A., Adversarial synthesis learning enables segmentation without target modality ground truth (2017) CoRR, abs/1712, p. 07695; Chang, Y.H., Burlingame, E.A., Gray, J.W., Margolin, A.A., SHIFT: speedy histopathological-to-immunofluorescent translation of whole slide images using conditional generative adversarial networks (2018) Medical Imaging 2018: Digital Pathology, , SPIE; Bentaieb, A., Hamarneh, G., Adversarial stain transfer for histopathology image analysis (2018) IEEE Trans. Med. Imaging, 37, pp. 792-802; Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A., (2017), Image-to-image translation with conditional adversarial networks, in Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR’17); Mirza, M., Osindero, S., Conditional generative adversarial nets (2014) arXiv, , 1411.1784; Zhu, J.-Y., Park, T., Isola, P., Efros, A.A., (2017), Unpaired image-to-image translation using cycle-consistent adversarial networks, in Proceedings of the International Conference on Computer Vision (ICCV’17); Koch, G., Zemel, R., Salakhutdinov, R., (2015), 2. , Siamese neural networks for one-shot image recognition, in Proceedings of the ICML Workshop on Deep Learning; Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P., (2016), pp. 2172-2180. , InfoGAN: interpretable representation learning by information maximizing generative adversarial nets, in Proceedings of the conference Advances in Neural Information Processing Systems; Levine, A.B., Peng, J., Farnell, D., Nursey, M., Wang, Y., Naso, J.R., Ren, H., Chiu, D., Synthesis of diagnostic quality cancer pathology images (2020) J. Pathol.; Karras, T., Aila, T., Laine, S., Lehtinen, J., (2018), Progressive growing of GANs for improved quality, stability, and variation, in Proceedings of the International Conference on Learning Representations (ICLR’18); Arjovsky, M., Chintala, S., Bottou, L., Wasserstein GAN (2017) arXiv, , 1701.07875; Hu, B., Tang, Y., Chang, E.I.-C., Fan, Y., Lai, M., Xu, Y., Unsupervised learning for cell-level visual representation in histopathology images with generative adversarial networks (2019) IEEE J. Biomed. Health Inform., 23, pp. 1316-1328; Lecouat, B., Chang, K., Foo, C.-S., Unnikrishnan, B., Brown, J.M., Zenati, H., Beers, A., Krishnaswamy, P., Semi-supervised deep learning for abnormality classification in retinal images (2018) arXiv, , 1812.07832; Zhou, N., Cai, D., Han, X., Yao, J., (2019), pp. 694-702. , Enhanced cycle-consistent generative adversarial network for color normalization of H&E stained images, in International Conference on Medical Image Computing and Computer-Assisted Intervention; Gadermayr, M., Gupta, L., Klinkhammer, B.M., Boor, P., Merhof, D., (2019), Unsupervisedly training GANs for segmenting digital pathology with automatically generated annotations, in Proceedings of the 2nd International Conference on Medical Imaging with Deep Learning (MIDL); Gupta, L., Klinkhammer, B.M., Boor, P., Merhof, D., Gadermayr, M., (2019), GAN-based image enrichment in digital pathology boosts segmentation accuracy, in Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI); Hou, L., Agarwal, A., Samaras, D., Kurç, T.M., Gupta, R.R., Saltz, J.H., Unsupervised histopathology image synthesis (2017) arXiv, , 1712.05021; Ghorbani, A., Natarajan, V., Coz, D., Liu, Y., Dermgan, Synthetic generation of clinical skin images with pathology (2019) arXiv, , 1911.08716; Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X., Improved techniques for training GANs (2016) Advances in Neural Information Processing Systems, pp. 2234-2242. , D.D. Lee M. Sugiyama U.V. Luxburg I. Guyon R. Garnett NIPS; Xu, Z., Moro, C.F., Bozóky, B., Zhang, Q., GAN-based virtual re-staining: a promising solution for whole slide image analysis (2019) arXiv, vol. abs/1901, p. 04059; Wang, D., Gu, C., Wu, K., Guan, X., (2017), Adversarial neural networks for basal membrane segmentation of microinvasive cervix carcinoma in histopathology images, in 2017 International Conference on Machine Learning and Cybernetics (ICMLC), IEEE; Almahairi, A., Rajeshwar, S., Sordoni, A., Bachman, P., Courville, A.C., (2018), Augmented cycleGAN: learning many-to-many mappings from unpaired data, in Proceedings of International Conference on Machine Learning (ICML’18); Huang, X., Liu, M.-Y., Belongie, S., Kautz, J., (2018), Multimodal unsupervised image-to-image translation, in Proceedings of the European Conference on Computer Vision (ECCV’18); Lahiani, A., Navab, N., Albarqouni, S., Klaiman, E., (2019), pp. 568-576. , Perceptual embedding consistency for seamless reconstruction of tilewise style transfer, in Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI’19); Modanwal, G., Vellal, A., Buda, M., Mazurowski, M.A., MRI image harmonization using cycle-consistent generative adversarial network (2020) Medical Imaging 2020: Computer-Aided Diagnosis, , H.K. Hahn M.A. Mazurowski SPIE; Gadermayr, M., Appel, V., Klinkhammer, B.M., Boor, P., Merhof, D., (2018), pp. 165-173. , Which way round? A study on the performance of stain-translation for segmenting arbitrarily dyed histological images, in Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI’18); Rana, A., Yauney, G., Lowe, A., Shah, P., (2018), Computational histological staining and destaining of prostate core biopsy RGB images with generative adversarial neural networks, in Proceedings of the IEEE International Conference on Machine Learning and Applications (ICMLA); Yang, J., Price, B., Cohen, S., Lee, H., Yang, M.-H., (2016), Object contour detection with a fully convolutional encoder-decoder network, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16), IEEE; Shrivastava, A., Gupta, A., Girshick, R., (2016), Training region-based object detectors with online hard example mining, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16), June; Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., Webb, R., (2017), Learning from simulated and unsupervised images through adversarial training, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Senaras, C., Niazi, M.K.K., Sahiner, B., Pennell, M.P., Tozbikian, G., Lozanski, G., Gurcan, M.N., Optimized generation of high-resolution phantom images using cGAN: application to quantification of Ki67 breast cancer images (2018) PLoS One, 13, p. e0196846; Gadermayr, M., Tschuchnig, M., Merhof, D., Krämer, N., Truhn, D., Gess, B., An asymmetric cycle-consistency loss for dealing with many-to-one mappings in image translation: a study on thigh MR scans (2020) CoRR, abs/2004, p. 11001; Wolterink, J.M., Dinkla, A.M., Savenije, M.H.F., Seevinck, P.R., (2017), pp. 14-23. , C.A.T. van den Berg, and I. Išgum, Deep MR to CT synthesis using unpaired data, in Proceedings of the International MICCAI Workshop Simulation and Synthesis in Medical Imaging (SASHIMI’17); Kearney, V., Ziemer, B.P., Perry, A., Wang, T., Chan, J.W., Ma, L., Morin, O., Solberg, T.D., Attention-aware discrimination for MR-to-CT image translation using cycle-consistent generative adversarial networks (2020) Radiol. Artif. Intelligence, 2, p. e190027; Lei, Y., Harms, J., Wang, T., Liu, Y., Shu, H.-K., Jani, A.B., Curran, W.J., Yang, X., MRI-only based synthetic CT generation using dense cycle consistent generative adversarial networks (2019) Med. Phys., 46, pp. 3565-3581; Gupta, A., Venkatesh, S., Chopra, S., Ledig, C., Generative image translation for data augmentation of bone lesion pathology (2019) arXiv, , 1902.02248; Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K., Spatial transformer networks (2015) Adv. Neural Inf. Process. Syst., 28, pp. 2017-2025","Tschuchnig, M.E.; Department of Information Technologies and Systems Management, Austria; email: maximilian.tschuchnig@fh-salzburg.ac.at",,,"Cell Press",,,,,26663899,,,,"English","Patterns",Review,"Final","All Open Access, Gold, Green",Scopus,2-s2.0-85097024499
"Abusitta A., Aïmeur E., Abdel Wahab O.","57254514300;6603130580;57215123692;","Generative adversarial networks for mitigating biases in machine learning systems",2020,"Frontiers in Artificial Intelligence and Applications","325",,,"937","944",,,"10.3233/FAIA200186","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85091750503&doi=10.3233%2fFAIA200186&partnerID=40&md5=ed6aa20da4a88c98418c54499528641d","McGill University, Canada; University of Montreal, Canada","Abusitta, A., McGill University, Canada; Aïmeur, E., McGill University, Canada; Abdel Wahab, O., University of Montreal, Canada","In this paper, we propose a new framework for mitigating biases in machine learning systems. The problem of the existing mitigation approaches is that they are model-oriented in the sense that they focus on tuning the training algorithms to produce fair results, while overlooking the fact that the training data can itself be the main reason for biased outcomes. Technically speaking, two essential limitations can be found in such model-based approaches: 1) the mitigation cannot be achieved without degrading the accuracy of the machine learning models, and 2) when the data used for training are largely biased, the training time automatically increases so as to find suitable learning parameters that help produce fair results. To address these shortcomings, we propose in this work a new framework that can largely mitigate the biases and discriminations in machine learning systems while at the same time enhancing the prediction accuracy of these systems. The proposed framework is based on conditional Generative Adversarial Networks (cGANs), which are used to generate new synthetic fair data with selective properties from the original data. We also propose a framework for analyzing data biases, which is important for understanding the amount and type of data that need to be synthetically sampled and labeled for each population group. Experimental results show that the proposed solution can efficiently mitigate different types of biases, while at the same time enhance the prediction accuracy of the underlying machine learning model. © 2020 The authors and IOS Press.",,"Population statistics; Adversarial networks; Learning parameters; Machine learning models; Model based approach; Prediction accuracy; Selective properties; Training algorithms; Training time; Machine learning",,,,,"Natural Sciences and Engineering Research Council of Canada, NSERC","The financial support of the Natural Sciences and Engineering Research Council of Canada is gratefully acknowledged. We also would like to acknowledge Dr. Gilles Brassard (University of Montreal), Dr. Kimiz Dalkir (McGill University), Younes Driouiche (Mila), Alexis Tremblay, Amine Belabed and Rim Ben Salem for helpful discussions.",,"(2019) The Adience Data Set, , https://talhassner.github.io/home/projects/Adience/Adience-data.html#agegender, (accessed April 2, 2019); (2019) Fairness and Bias of the COMPAS Algorithm Compared to Human Assessments, , https://www.hiit.fi/, (accessed November 10, 2019); (2019) COMPAS Recidivism Risk Score Data and Analysis, , https://www.propublica.org/, (accessed November 15, 2019); (2019) Amazon Rekognition, , https://aws.amazon.com/rekognition/nc1=h_ls, (accessed November 17, 2019); (2019) Google Photos, , https://play.google.com/store/apps, (accessed November 17, 2019); (2019) Machine Learning and Bias, , https://developer.ibm.com/articles/machine-learning-and-bias/, (accessed November 17, 2019); Abusitta, A., Bellaiche, M., Dagenais, M., On trustworthy federated clouds: A coalitional game approach (2018) Computer Networks, 145, pp. 52-63; Abusitta, A., Bellaiche, M., Dagenais, M., An SVM-based framework for detecting dos attacks in virtualized clouds under changing environment (2018) Journal of Cloud Computing, 7 (1), p. 9; Abusitta, A., Bellaiche, M., Dagenais, M., A trust-based game theoretical model for cooperative intrusion detection in multicloud environments (2018) 2018 21st Conference on Innovation in Clouds, Internet and Networks and Workshops (ICIN), pp. 1-8. , IEEE; Abusitta, A., Bellaiche, M., Dagenais, M., Halabi, T., A deep learning approach for proactive multi-cloud cooperative intrusion detection system (2019) Future Generation Computer Systems; Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., Wallach, H., (2018) A Reductions Approach to Fair Classification; Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H., Greedy layer-wise training of deep networks (2007) Advances in Neural Information Processing Systems, pp. 153-160; Brackey, A., (2019) Analysis of Racial Bias in Northpointe's COMPAS Algorithm, , Ph. D. Dissertation, Tulane University School of Science and Engineering; Calmon, F., Wei, D., Vinzamuri, B., Natesan Ramamurthy, K., Varshney, K.R., Optimized pre-processing for discrimination prevention (2017) Advances in Neural Information Processing Systems, pp. 3992-4001; Camino, R., Hammerschmidt, C., State, R., (2018) Generating Multi-categorical Samples with Generative Adversarial Networks; Campolo, A., Sanfilippo, M., Whittaker, M., Crawford, K., Ai now 2017 report (2017) AI Now Institute at New York University; Elisa Celis, L., Keswani, V., (2019) Improved Adversarial Learning for Fair Classification; Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., Tsaneva-Atanasova, K., Artificial intelligence, bias and clinical safety (2019) BMJ Qual Saf, 28 (3), pp. 231-237; Chen, X., Wang, J., Ge, H., (2018) Training Generative Adversarial Networks Via Primal-dual Subgradient Methods: A Lagrangian Perspective on Gan; Doersch, C., (2016) Tutorial on Variational Autoencoders; Feldman, M., Friedler, S.A., Moeller, J., Scheidegger, C., Venkatasubramanian, S., Certifying and removing disparate impact (2015) Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 259-268. , ACM; Goh, G., Cotter, A., Gupta, M., Friedlander, M.P., Satisfying real-world goals with dataset constraints (2016) Advances in Neural Information Processing Systems, pp. 2415-2423; Goodfellow, I., (2016) Nips 2016 Tutorial: Generative Adversarial Networks; Goodfellow, I., Bengio, Y., Courville, A., (2016) Deep Learning, , MIT press; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Advances in Neural Information Processing Systems, pp. 2672-2680; Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C., Improved training of wasserstein gans (2017) Advances in Neural Information Processing Systems, pp. 5767-5777; Halabi, T., Bellaiche, M., Abusitta, A., A cooperative game for online cloud federation formation based on security risk assessment (2018) 2018 5th IEEE International Conference on Cyber Security and Cloud Computing (CSCloud)/2018 4th IEEE International Conference on Edge Computing and Scalable Cloud (EdgeCom), pp. 83-88. , IEEE; Halabi, T., Bellaiche, M., Abusitta, A., Toward secure resource allocation in mobile cloud computing: A matching game (2019) 2019 International Conference on Computing, Networking and Communications (ICNC), pp. 370-374. , IEEE; Hardt, M., Price, E., Srebro, N., Equality of opportunity in supervised learning (2016) Advances in Neural Information Processing Systems, pp. 3315-3323; Hinton, G.E., Osindero, S., Teh, Y.-W., A fast learning algorithm for deep belief nets (2006) Neural Computation, 18 (7), pp. 1527-1554; Jang, E., Gu, S., Poole, B., (2016) Categorical Reparameterization with Gumbel-softmax; Kamiran, F., Calders, T., Data preprocessing techniques for classification without discrimination (2012) Knowledge and Information Systems, 33 (1), pp. 1-33; Karani, D., (2019) Introduction to Word Embedding and Word2Vec, , https://towardsdatascience.com/, (accessed November 17, 2019); Kenney, M., (2019) Amazon Rekognition, , http://matthewkenney.site/biases.html, (accessed November 17, 2019); Khosla, A., Zhou, T., Malisiewicz, T., Efros, A.A., Torralba, A., Undoing the damage of dataset bias (2012) European Conference on Computer Vision, pp. 158-171. , Springer; Kingma, D.P., Mohamed, S., Jimenez Rezende, D., Welling, M., Semi-supervised learning with deep generative models (2014) Advances in Neural Information Processing Systems, pp. 3581-3589; Kivinen, J., Warmuth, M.K., Exponentiated gradient versus gradient descent for linear predictors (1997) Information and Computation, 132 (1), pp. 1-63; Krasanakis, E., Spyromitros-Xioufis, E., Papadopoulos, S., Kompatsiaris, Y., Adaptive sensitive reweighting to mitigate bias in fairness-aware classification (2018) Proceedings of the 2018 World Wide Web Conference on World Wide Web, pp. 853-862. , InternationalWorldWideWeb Conferences Steering Committee; LeCun, Y., Bengio, Y., Hinton, G., Deep learning (2015) Nature, 521 (7553), p. 436; Louppe, G., Kagan, M., Cranmer, K., Learning to pivot with adversarial networks (2017) Advances in Neural Information Processing Systems, pp. 981-990; Maddison, C.J., Mnih, A., Whye Teh, Y., (2016) The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables; Madras, D., Creager, E., Pitassi, T., Zemel, R., (2018) Learning Adversarially Fair and Transferable Representations; Mirza, M., Osindero, S., (2014) Conditional Generative Adversarial Nets; Mitchell, T.M., (1980) The Need for Biases in Learning Generalizations, , Department of Computer Science, Laboratory for Computer Science Research; O'Neill, B., Nonmetric test of the minimax theory of two-person zerosum games (1987) Proceedings of the National Academy of Sciences, 84 (7), pp. 2106-2109; Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J., Weinberger, K.Q., On fairness and calibration (2017) Advances in Neural Information Processing Systems, pp. 5680-5689; Rodríguez, P., Cucurull, G., Gonfaus, J.M., Xavier Roca, F., Gonzalez, J., Age and gender recognition in the wild with deep attention (2017) Pattern Recognition, 72, pp. 563-571; Ruder, S., Peters, M.E., Swayamdipta, S., Wolf, T., Transfer learning in natural language processing (2019) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pp. 15-18; Singhi, S.K., Liu, H., Feature subset selection bias for classification learning (2006) Proceedings of the 23rd International Conference on Machine Learning, pp. 849-856; Dieterich, W., Brennan, T., (2019) Correctional Offender Management Profiles for Alternative Sanctions (COMPAS), , https://www.researchgate.net/publication/321528262_Correctional_Offender_Management_Profiles_for_Alternative_Sanctions_COMPAS, (accessed November 10, 2019); Tonk, S., (2019) Towards Fairness in ML with Adversarial Networks, , https://blog.godatadriven.com/fairness-in-ml, (accessed April 2, 2019); Torralba, A., Efros, A.A., Unbiased look at dataset bias (2011) CVPR 2011, pp. 1521-1528. , IEEE; Wakabayashi, D., (2019) Google Finds It's Underpaying Many Men As It Addresses Wage Equity, , https://www.nytimes.com/2019/03/04/technology/google-gender-pay-gap.html, (accessed May 2, 2019); Woodworth, B., Gunasekar, S., Ohannessian, M.I., Srebro, N., (2017) Learning Non-discriminatory Predictors; Xu, D., Yuan, S., Zhang, L., Wu, X., Fairgan: Fairnessaware generative adversarial networks (2018) 2018 IEEE International Conference on Big Data (Big Data), pp. 570-575. , IEEE; Yen, S.-J., Lee, Y.-S., Under-sampling approaches for improving prediction of the minority class in an imbalanced dataset (2006) Intelligent Control and Automation, pp. 731-740. , Springer; Hu Zhang, B., Lemoine, B., Mitchell, M., Mitigating unwanted biases with adversarial learning (2018) Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335-340. , ACM",,"De Giacomo G.Catala A.Dilkina B.Milano M.Barro S.Bugarin A.Lang J.","Accenture;Artificial Intelligence;et al.;Hewlett Packard;iecisa IBM;Intel","IOS Press BV","24th European Conference on Artificial Intelligence, ECAI 2020, including 10th Conference on Prestigious Applications of Artificial Intelligence, PAIS 2020","29 August 2020 through 8 September 2020",,162625,09226389,9781643681009,,,"English","Front. Artif. Intell. Appl.",Conference Paper,"Final","",Scopus,2-s2.0-85091750503
"Du Q., Qiang Y., Yang W., Wang Y., Ma Y., Zia M.B.","57212001911;26639724500;57211025809;57221509900;57200728049;57212009352;","DRGAN: A deep residual generative adversarial network for PET image reconstruction",2020,"IET Image Processing","14","9",,"1851","1861",,3,"10.1049/iet-ipr.2019.1107","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85089876607&doi=10.1049%2fiet-ipr.2019.1107&partnerID=40&md5=cdecb5c33dc428fe6a2f28e3df6b1298","College of Information and Computer, Taiyuan University of Technology, Taiyuan, 030024, China; Department of Thoracic Surgery, Shanxi Cancer Hospital, Taiyuan, 030024, China","Du, Q., College of Information and Computer, Taiyuan University of Technology, Taiyuan, 030024, China; Qiang, Y., College of Information and Computer, Taiyuan University of Technology, Taiyuan, 030024, China; Yang, W., College of Information and Computer, Taiyuan University of Technology, Taiyuan, 030024, China; Wang, Y., College of Information and Computer, Taiyuan University of Technology, Taiyuan, 030024, China; Ma, Y., Department of Thoracic Surgery, Shanxi Cancer Hospital, Taiyuan, 030024, China; Zia, M.B., College of Information and Computer, Taiyuan University of Technology, Taiyuan, 030024, China","Positron emission tomography (PET) image reconstruction from low-count projection data and physical effects is challenging because the inverse problem is ill-posed and the resultant image is usually noisy. Recently, generative adversarial networks (GANs) have also shown their superior performance in many computer vision tasks and attracted growing interests in medical imaging. In this work, the authors proposed a novel model [deep residual generative adversarial network (DRGAN)] based on GANs for the reduction of streaking artefacts and the improvement of PET image quality. An innovative feature of the proposed method is that the authors trained a generator to produce 'residual PET map' (RPM) for image representation, rather than generate PET images directly. DRGAN used two discriminators (critics) to enforce anatomically realistic PET images and RPM. To better boost the contextual information, the authors designed residual dense connections followed with pixel shuffle operations (RDPS blocks) that encourage feature reuse and prevent losing resolution. Both simulation data and real clinical PET data are used to evaluate the proposed method. Compared with other state-of-the-art methods, the quantification results show that DRGAN can achieve better performance in bias-variance trade-off and provide comparable image quality. Their results were rigorously evaluated by one radiologist at the Shanxi Cancer Hospital. © The Institution of Engineering and Technology 2020",,"Economic and social effects; Image enhancement; Image quality; Inverse problems; Medical imaging; Positron emission tomography; Adversarial networks; Bias variance trade off; Contextual information; Image representations; PET image reconstruction; Physical effects; Positron emission tomography (PET); State-of-the-art methods; Image reconstruction",,,,,"National Natural Science Foundation of China, NSFC: 201801D121139, 61872261

Fujian Provincial Department of Science and Technology","This work was supported by the National Natural Science Foundation of China (NSFC) under grant number 61872261 and the basic research (201801D121139, Development of Novel Artificial Intelligence Technologies to Assist Imaging Diagnosis of Pulmonary) funded by the Provincial Department of Science and Technology (Shanxi, China).",,"Wang, Y., Zhou, L., Wang, L., Locality adaptive multi-modality GANs for high-quality pet image synthesis (2018) Int. Conf. On Medical Image Computing and Computer-Assisted Intervention, pp. 329-337. , Granada, Spain; Karp, J.S., Surti, S., Daube-Witherspoon, M.E., Benefit of time of-flight in pet: Experimental and clinical results (2008) J. Nucl. Med., 49 (3), pp. 462-470; Yang, Y., Qi, J., Wu, Y., Depth of interaction calibration for pet detectors with dual-ended readout by psapds (2008) Phys. Med. Biol., 54 (2), p. 433; Poon, J.K., Dahlbom, M.L., Moses, W.W., Optimal whole-body pet scanner configurations for different volumes of lso scintillator: A simulation study (2012) Phys. Med. Biol., 57 (13), p. 4077; Gong, K., Majewski, S., Kinahan, P.E., Designing a compact high performance brain pet scanner - Simulation study (2016) Phys. Med. Biol., 61 (10), p. 3681; Hunt, B.R., Image reconstruction from projections: Implementation and applications (1981) J. Siam Review, 23 (3), pp. 142-149; Lange, K., Carson, R., Em reconstruction algorithms for emission and transmission tomography (1984) J. Comput. Assist. Tomogr., 8 (2), pp. 306-316; Wang, C., Hu, Z., Shi, P., Low dose pet reconstruction with total variation regularization (2014) 2014 36th Annual Int. Conf. Of the IEEE Engineering in Medicine and Biology Society, pp. 1917-1920. , Chicago, IL, USA; Buades, A., Coll, B., Morel, J.M., A non-local algorithm for image denoising (2005) 2005 IEEE Computer Society Conf. On Computer Vision and Pattern Recognition (CVPR'05), 2, pp. 60-65. , San Diego, CA, USA; Dabov, K., Foi, A., Katkovnik, V., Image denoising with blockmatching and 3d filtering (2006) Image Processing: Algorithms and Systems, Neural Networks, and Machine Learning, 6064, p. 606414. , San Jose, CA, USA; Yu, S., Muhammed, H.H., Noise type evaluation in positron emission tomography images (2016) 2016 1st Int. Conf. On Biomedical Engineering (IBIOMED), pp. 1-6. , Yogyakarta, Indonesia; Dutta, J., Leahy, R.M., Li, Q., Non-local means denoising of dynamic pet images (2013) PLoS ONE, 8 (12); Bagci, U., Mollura, D.J., Denoising pet images using singular value thresholding and Stein's unbiased risk estimate (2013) Int. Conf. On Medical Image Computing and Computer-Assisted Intervention, pp. 115-122. , Nagoya, Japan; Wang, G., Kalra, M., Orton, C.G., Machine learning will transform radiology significantly within the next 5 years (2017) Med. Phys., 44 (6), pp. 2041-2044; Wang, G., A perspective on deep imaging (2016) IEEE Access, 4, pp. 8914-8924; Gong, K., Guan, J., Kim, K., Iterative pet image reconstruction using convolutional neural network representation (2018) IEEE Trans. Med. Imaging, 38 (3), pp. 675-685; Kim, K., Wu, D., Gong, K., Penalized pet reconstruction using deep learning prior and local linear fitting (2018) IEEE Trans. Med. Imaging, 37 (6), pp. 1478-1487; Zhang, K., Zuo, W., Chen, Y., Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising (2017) IEEE Trans Image Process, 26 (7), pp. 3142-3155; Jiao, J., Ourselin, S., (2017) Fast Pet Reconstruction Using Multi-Scale Fully Convolutional Neural Networks, , preprint; Chen, H., Zhang, Y., Kalra, M.K., Low-dose ct with a residual encoder-decoder convolutional neural network (2017) IEEE Trans. Med. Imaging, 36 (12), pp. 2524-2535; Gong, K., Guan, J., Liu, C.C., Pet image denoising using a deep neural network through fine tuning (2018) IEEE Trans. Radiat. Plasma Med.Sci., 3 (2), pp. 153-161; Ledig, C., Theis, L., Huszár, F., Photo-realistic single image super-resolution using a generative adversarial network (2017) Proc. Of the IEEE Conf. Of Computer Vision and Pattern Recognition, pp. 4681-4690. , Puerto Rico, USA; Qiao, J., Song, H., Zhang, K., Image super-resolution using conditional generative adversarial network (2019) IET Image Process, 13 (14), pp. 2673-2679; Brock, A., Lim, T., Ritchie, J.M., (2016) Neural Photo Editing with Introspective Adversarial Networks, , preprint; Zhu, J.Y., Krähenbühl, P., Shechtman, E., Generative visual manipulation on the natural image manifold (2016) European Conf. On Computer Vision, pp. 597-613. , Amsterdam, The Netherlands; Isola, P., Zhu, J.Y., Zhou, T., Image-to-image translation with conditional adversarial networks (2017) Proc. Of the IEEE Conf. Of Computer Vision and Pattern Recognition, pp. 1125-1134. , Honolulu, HI, USA; Wang, J., Li, J., Sun, B., SAR image synthesis based on conditional generative adversarial networks (2019) J. Eng., 2019 (21), pp. 8093-8097; Nie, D., Trullo, R., Lian, J., Medical image synthesis with context-aware generative adversarial networks (2017) Int. Conf. On Medical Image Computing and Computer-Assisted Intervention, pp. 417-425. , Quebec City, QC, Canada; Wolterink, J.M., Leiner, T., Viergever, M.A., Generative adversarial networks for noise reduction in low-dose ct (2017) IEEE Trans. Med. Imaging, 36 (12), pp. 2536-2545; Yang, G., Yu, S., Dong, H., Dagan: Deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction (2017) IEEE Trans. Med. Imaging, 37 (6), pp. 1310-1321; Kodali, N., Abernethy, J., Hays, J., (2017) On Convergence and Stability of Gans, , preprint; Arjovsky, M., Bottou, L., (2017) Towards Principled Methods for Training Generative Adversarial Networks, , preprint; He, K., Zhang, X., Ren, S., Deep residual learning for image recognition (2016) Proc. Of the IEEE Conf. Of Computer Vision and Pattern Recognition, pp. 770-778. , Las Vegas, NV, USA; Radford, A., Metz, L., Chintala, S., (2015) Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, , preprint; Arjovsky, M., Chintala, S., Bottou, L., (2017) Wasserstein Gan, , preprint; Humm, J.L., Rosenfeld, A., Del-Guerra, A., From pet detectors to pet scanners (2003) Eur. J. Nucl. Med. Mol. Imaging, 30 (11), pp. 1574-1597; Kim, K., Dutta, J., Groll, A., Penalized maximum likelihood reconstruction of ultrahigh resolution pet with depth of interaction (2015) The 13th Int. Meeting on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine, pp. 296-299. , Newport, RI, USA; Wang, G., Qi, J., Penalized likelihood pet image reconstruction using patch-based edge-preserving regularization (2012) IEEE Trans. Med. Imaging, 31 (12), pp. 2194-2204; Branderhorst, W., Vastenhouw, B., Beekman, F.J., Pixel-based subsets for rapid multi-pinhole spect reconstruction (2010) Phys. Med. Biol., 55 (7), p. 2023; Panin, V.Y., Ordered subsets acceleration of iterative algorithm for variance reduction on compressed sinogram random coincidences (2011) 2011 IEEE Nuclear Science Symp. Conf. Record, pp. 2986-2990. , Valencia, Spain; Chen, H., Zhang, Y., Zhang, W., Low-dose CT via convolutional neural network (2017) Biomed. Opt. Express, 8 (2), pp. 679-694; Portilla, J., Strela, V., Wainwright, M.J., Image denoising using scale mixtures of gaussians in the wavelet domain (2003) IEEE Trans. Image Process., 12 (11), pp. 1338-1351; Le-Pogam, A., Hanzouli, H., Hatt, M., Denoising of pet images by combining wavelets and curvelets for improved preservation of resolution and quantitation (2013) Med. Image Anal., 17 (8), pp. 877-891; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Generative adversarial nets (2014) Advances in Neural Information Processing Systems, pp. 2672-2680. , Montreal, Canada; Ronneberger, O., Fischer, P., Brox, T., U-Net: Convolutional networks for biomedical image segmentation (2015) Int. Conf. On Medical Image Computing and Computer-Assisted Intervention, pp. 234-241. , Munich, Germany; Huang, G., Liu, Z., van der Maaten, L., Densely connected convolutional networks (2017) Proc. Of the IEEE Conf. Of Computer Vision and Pattern Recognition, pp. 4700-4708. , Honolulu, HI, USA; Yu, F., Koltun, V., (2015) Multi-Scale Context Aggregation by Dilated Convolutions, , preprint; Noh, H., Hong, S., Han, B., Learning deconvolution network for semantic segmentation (2015) Proc. Of the IEEE Int. Conf. Of Computer Vision, pp. 1520-1528. , Santiago, Chile; Shi, W., Caballero, J., Huszár, F., Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network (2016) Proc. Of the IEEE Conf. Of Computer Vision and Pattern Recognition, pp. 1874-1883. , Las Vegas, NV, USA; Szegedy, C., Liu, W., Jia, Y., Going deeper with convolutions (2015) Proc. Of the IEEE Conf. Of Computer Vision and Pattern Recognition, pp. 1-9. , Boston, MA, USA; Johnson, J., Alahi, A., Fei-Fei, L., Perceptual losses for real-time style transfer and super-resolution (2016) European Conf. On Computer Vision, pp. 694-711. , Amsterdam, The Netherlands; Yang, Q., Yan, P., Kalra, M.K., (2017) Ct Image Denoising with Perceptive Deep Neural Networks, , preprint; Simonyan, K., Zisserman, A., (2014) Very Deep Convolutional Networks for Large-Scale Image Recognition, , preprint; Russakovsky, O., Deng, J., Su, H., Imagenet large scale visual recognition challenge (2015) Int. J. Comput. Vis., 115 (3), pp. 211-252; He, K., Zhang, X., Ren, S., Delving deep into rectifiers: Surpassing human-level performance on imagenet classification (2015) Proc. Of the IEEE Int. Conf. Of Computer Vision, pp. 1026-1034. , Santiago, Chile; Kingma, D.P., Ba, J., (2014) Adam: A Method for Stochastic Optimization, , preprint; Huang, X., Li, Y., Poursaeed, O., Stacked generative adversarial networks (2017) Proc. Of the IEEE Conf. Of Computer Vision and Pattern Recognition, pp. 5077-5086. , Honolulu, HI, USA; Simard, P.Y., Steinkraus, D., Platt, J.C., Best practices for convolutional neural networks applied to visual document analysis (2003) ICDAR, 3. , Scotland, UK; Dong, H., Supratak, A., Mai, L., Tensorlayer: A versatile library for efficient deep learning development (2017) Proc. Of the 25th ACM Int. Conf. Of Multimedia, pp. 1201-1204. , Mountain View, CA, USA; Aharon, M., Elad, M., Bruckstein, A., K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation (2006) IEEE Trans. Signal Process., 54 (11), pp. 4311-4322; Socher, R., Ganjoo, M., Manning, C.D., Zero-shot learning through crossmodal transfer (2013) Advances in Neural Information Processing Systems, pp. 935-943. , Lake Tahoe, NV, USA","Qiang, Y.; College of Information and Computer, China; email: qiangyan@tyut.edu.cn",,,"Institution of Engineering and Technology",,,,,17519659,,,,"English","IET Image Proc.",Article,"Final","All Open Access, Bronze",Scopus,2-s2.0-85089876607
"Chui K.T., Lytras M.D., Vasant P.","55975707100;55830169000;8948616300;","Combined generative adversarial network and fuzzy C-means clustering for multi-class voice disorder detection with an imbalanced dataset",2020,"Applied Sciences (Switzerland)","10","13","4571","","",,5,"10.3390/app10134571","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85087832634&doi=10.3390%2fapp10134571&partnerID=40&md5=66cf0fd9663ed6bfac8cd881609267e2","School of Science and Technology, The Open University of Hong Kong, Hong Kong; King Abdulaziz University, P.O. Box 34689, Jeddah, Saudi Arabia; Effat College of Engineering, Effat University, P.O. Box 34689, Jeddah, Saudi Arabia; Fundamental and Applied Sciences Department, Universiti Teknologi PETRONAS, Seri Iskandar, Perak Darul Ridzuan, 32610, Malaysia","Chui, K.T., School of Science and Technology, The Open University of Hong Kong, Hong Kong; Lytras, M.D., King Abdulaziz University, P.O. Box 34689, Jeddah, Saudi Arabia, Effat College of Engineering, Effat University, P.O. Box 34689, Jeddah, Saudi Arabia; Vasant, P., Fundamental and Applied Sciences Department, Universiti Teknologi PETRONAS, Seri Iskandar, Perak Darul Ridzuan, 32610, Malaysia","The world has witnessed the success of artificial intelligence deployment for smart healthcare applications. Various studies have suggested that the prevalence of voice disorders in the general population is greater than 10%. An automatic diagnosis for voice disorders via machine learning algorithms is desired to reduce the cost and time needed for examination by doctors and speech-language pathologists. In this paper, a conditional generative adversarial network (CGAN) and improved fuzzy c-means clustering (IFCM) algorithm called CGAN-IFCM is proposed for the multi-class voice disorder detection of three common types of voice disorders. Existing benchmark datasets for voice disorders, the Saarbruecken Voice Database (SVD) and the Voice ICar fEDerico II Database (VOICED), use imbalanced classes. A generative adversarial network offers synthetic data to reduce bias in the detection model. Improved fuzzy c-means clustering considers the relationship between adjacent data points in the fuzzy membership function. To explain the necessity of CGAN and IFCM, a comparison is made between the algorithm with CGAN and that without CGAN. Moreover, the performance is compared between IFCM and traditional fuzzy c-means clustering. Lastly, the proposed CGAN-IFCM outperforms existing models in its true negative rate and true positive rate by 9.9-12.9% and 9.1-44.8%, respectively. © 2020 by the authors.","Artificial intelligence; Fuzzy c-means clustering; Generative adversarial network; Imbalanced dataset; Machine learning; Multi-class detection; Smart healthcare; Synthetic data; Voice disorders",,,,,,,,,"Vilkman, E., Voice problems at work: A challenge for occupational safety and health arrangement (2000) Folia Phoniatrica et Logopaedica, 52, pp. 120-125; Dodderi, T., Philip, N.E., Mutum, K., Prevalence of voice disorders in the Department of Speech Language Pathology of a tertiary care hospital of Mangaluru: A retrospective study of 11 years (2018) Nitte Univ. J. Health Sci, 8, pp. 12-16; Lyberg-Ahlander, V., Rydell, R., Fredlund, P., Magnusson, C., Wilén, S., Prevalence of voice disorders in the general population, based on the Stockholm public health cohort (2019) J. Voice, 33, pp. 900-905; Vertanen-Greis, H., Löyttyniemi, E., Uitti, J., Voice disorders are associated with stress among teachers: A cross-sectional study in Finland (2018) J. Voice, 34, pp. 488e1-488e8; Roy, N., Merrill, R.M., Gray, S.D., Smith, E.M., Voice disorders in the general population: Prevalence, risk factors, and occupational impact (2005) Laryngoscope, 115, pp. 1988-1995; Leào, S.H.D.S., Oates, J.M., Purdy, S.C., Scott, D., Morton, R.P., Voice problems in New Zealand teachers: A national survey (2015) J. Voice, 29; Muhammad, G., Alhamid, M.F., Alsulaiman, M., Gupta, B., Edge computing with cloud for voice disorder assessment and treatment (2018) IEEE Commun. Mag, 56, pp. 60-65; Alhussein, M., Muhammad, G., Voice pathology detection using deep learning on mobile healthcare framework (2018) IEEE Access, 6, pp. 41034-41041; Amami, R., Amami, R., Eleraky, H.A., An Incremental System for Voice Pathology Detection Combining Possibilistic SVM and HMM (2019) Proceedings of the International Conference on Statistical Language and Speech Processing, pp. 127-138. , Ljubljana, Slovenia, 14-16 October; Fang, S.H., Tsao, Y., Hsiao, M.J., Chen, J.Y., Lai, Y.H., Lin, F.C., Wang, C.T., Detection of pathological voice using cepstrum vectors: A deep learning approach (2019) J. Voice, 33, pp. 634-641; Ali, Z., Imran, M., Alsulaiman, M., Zia, T., Shoaib, M., A zero-watermarking algorithm for privacy protection in biomedical signals (2018) Future Gener. Comput. Syst, 82, pp. 290-303; Amara, F., Fezari, M., Bourouba, H., An improved GMM-SVM system based on distance metric for voice pathology detection (2016) Appl. Math, 10, pp. 1061-1070; Verde, L., de Pietro, G., Sannino, G., Voice disorder identification by using machine learning techniques (2018) IEEE Access, 6, pp. 16246-16255; Guedes, V., Teixeira, F., Oliveira, A., Fernandes, J., Silva, L., Junior, A., Teixeira, J.P., Transfer Learning with AudioSet to Voice Pathologies Identification in Continuous Speech (2019) Procedia Comput. Sci, 164, pp. 662-669; Kadiri, S.R., Alku, P., Analysis and Detection of Pathological Voice using Glottal Source Features (2020) IEEE J. Sel. Top. Signal Process, 14, pp. 367-379; Verde, L., de Pietro, G., Alrashoud, M., Ghoneim, A., Al-Mutib, K.N., Sannino, G., Dysphonia Detection Index (DDI): A New Multi-Parametric Marker to Evaluate Voice Quality (2019) IEEE Access, 7, pp. 55689-55697; Chen, L., Wang, C., Chen, J., Xiang, Z., Hu, X., Voice Disorder Identification by using Hilbert-Huang Transform (HHT) and K Nearest Neighbor (KNN) (2020) J. Voice; Verde, L., de Pietro, G., Alrashoud, M., Ghoneim, A., Al-Mutib, K.N., Sannino, G., Leveraging Artificial Intelligence to Improve Voice Disorder Identification Through the Use of a Reliable Mobile App (2019) IEEE Access, 7, pp. 124048-124054; Pützer, M., Koreman, J., A German database of patterns of pathological vocal fold vibration (1997) Phonus, 3, pp. 143-153; http://www.stimmdatenbank.coli.uni-saarland.de/help_en.php4, (accessed on 20 February 2020); Cesari, U., de Pietro, G., Marciano, E., Niri, C., Sannino, G., Verde, L., A new database of healthy and pathological voices (2018) Comput. Elect. Eng, 68, pp. 310-321; Pan, Z., Yu, W., Yi, X., Khan, A., Yuan, F., Zheng, Y., Recent progress on generative adversarial networks (GANs): A survey (2019) IEEE Access, 7, pp. 36322-36333; Odena, A., Olah, C., Shlens, J., Conditional image synthesis with auxiliary classifier gans (2017) Proceedings of the 34th International Conference on Machine Learning, pp. 2642-2651. , Sydney, Australia, 6-11 August; Mirza, M., Osindero, S., Conditional Generative Adversarial Nets, , https://arxiv.org/abs/1411.1784, (accessed on 10 April 2020); Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P., Infogan: Interpretable representation learning by information maximizing generative adversarial nets (2016) Proceedings of the Advances in Neural Information Processing Systems, pp. 2172-2180. , Barcelona, Spain, 5-10 December; Brockmann, M., Drinnan, M.J., Storck, C., Carding, P.N., Reliable jitter and shimmer measurements in voice clinics: The relevance of vowel, gender, vocal intensity, and fundamental frequency effects in a typical clinical task (2011) J. Voice, 25, pp. 44-53; Lopes, L.W., da Silva, J.D., Simöes, L.B., da Silva Evangelista, D., Silva, P.O.C., Almeida, A.A., de Lima-Silva, M.F.B., Relationship between acoustic measurements and self-evaluation in patients with voice disorders (2017) J. Voice, 31, pp. 119e1-119e10; Severin, F., Bozkurt, B., Dutoit, T., HNR extraction in voiced speech, oriented towards voice quality analysis (2005) Proceedings of the 2005 13th European Signal Processing Conference, pp. 1-4. , Antalya, Turkey, 4-8 September; Farrús, M., Hernando, J., Ejarque, P., Jitter and shimmer measurements for speaker recognition (2007) Proceedings of the Eighth Annual Conference of the International Speech Communication Association, pp. 778-781. , Antwerp, Belgium, August 27-31; Verde, L., de Pietro, G., Sannino, G., A methodology for voice classification based on the personalized fundamental frequency estimation (2018) Biomed. Signal Process. Control, 42, pp. 134-144; Grimm, K.J., Mazza, G.L., Davoudzadeh, P., Model selection in finite mixture models: A k-fold cross-validation approach (2017) Struct. Equ. Model, 24, pp. 246-256; Varoquaux, G., Raamana, P.R., Engemann, D.A., Hoyos-Idrobo, A., Schwartz, Y., Thirion, B., Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines (2017) NeuroImage, 145, pp. 166-179; Bezdek, J.C., (1981) Pattern Recognition with Fuzzy Objective Function Algorithms;, , Kluwer Academic Publishers: Norwell, MA, USA; Maulik, U., Bandyopadhyay, S., Performance evaluation of some clustering algorithms and validity indices (2002) IEEE Trans. Pattern Anal. Mach. Intell, 24, pp. 1650-1654; Foneseca, C.M., Flemming, P., Genetic algorithms for multi-objective optimization: Formulation, discussion, and generalization (1993) Proceedings of the 5th International Conference on Genetic Algorithms, pp. 416-423. , Urbana-Champaign, Champaign, IL, USA, 17-21 July 1993; Morgan Kaufmann: San Francisco, CA, USA; Deb, K., (2001) Multi-Objective Optimization Using Evolutionary Algorithms;, , John Wiley & Sons, Inc.: New York, NY, USA; Jensen, M.T., Reducing the run-time complexity of multiobjective EAs: The NSGA-II and other algorithms (2003) IEEE Trans. Evol. Comput, 7, pp. 503-515; Dutta, S., Das, K.N., A survey on pareto-based eas to solve multi-objective optimization problems (2019) Soft Computing for Problem Solving;, , Bansal, J., Das, K., Nagar, A., Deep, K., Ojha, A., Eds.; Advances in Intelligent Systems and Computing; Springer: Singapore; Goldberg, D., Richardson, J., Genetic Algorithms with Sharing for Multi-modal Function Optimization (1987) Proceedings of the International Conference on Genetic Algorithms, pp. 41-49. , Cambridge, MA, USA, 28-31 July; Mahfoud, S.W., (1995) Niching Methods for Genetic Algorithms, , Ph.D. Thesis, University of Illinois at Urbana-Champaign, Urbana Champaign, IL, USA; Ji, J.Y., Yu, W.J., Zhong, J., Zhang, J., Density-Enhanced Multiobjective Evolutionary Approach for Power Economic Dispatch Problems (2019) IEEE Trans. Syst. Man Cybern. Syst; Maldonado, S., López, J., Vairetti, C., An alternative SMOTE oversampling strategy for high-dimensional datasets (2019) Appl. Soft Comput, 76, pp. 380-389; Sun, J., Li, H., Fujita, H., Fu, B., Ai, W., Class-imbalanced dynamic financial distress prediction based on Adaboost-SVM ensemble combined with SMOTE and time weighting (2020) Inf. Fusion, 54, pp. 128-144; Jia, X., Li, W., Shang, L., A multiphase cost-sensitive learning method based on the multiclass three-way decision-theoretic rough set model (2019) Inf. Sci, 485, pp. 248-262; Feng, F., Li, K.C., Shen, J., Zhou, Q., Yang, X., Using cost-sensitive learning and feature selection algorithms to improve the performance of imbalanced classification (2020) IEEE Access, 8, pp. 69979-69996; Limpert, E., Stahel, W.A., Problems with using the normal distribution-and ways to improve quality and efficiency of data analysis (2011) PLoS ONE, 6; de Winter, J.C., Using the Student's t-test with extremely small sample sizes (2013) Pract. Assess. Res. Eval, 18, p. 10; Ngyen, K.A., Chen, W., Lin, B.S., Seeboonruang, U., Using Machine Learning-Based Algorithms to Analyze Erosion Rates of a Watershed in Northern Taiwan (2020) Sustainability, 12, p. 2022; Meek, G.E., Ozgur, C., Dunning, K., Comparison of the t vs. Wilcoxon signed-rank test for Likert scale data and small samples (2007) J. Mod. Appl. Stat. Methods, 6, p. 10","Chui, K.T.; School of Science and Technology, Hong Kong; email: jktchui@ouhk.edu.hk",,,"MDPI AG",,,,,20763417,,,,"English","Appl. Sci.",Article,"Final","All Open Access, Gold",Scopus,2-s2.0-85087832634
"Bao S., Wang Z., Liu T., Chen D., Cai Y., Huang R.","57211713570;56266980600;57211848884;57221476657;55550414100;57211065268;","Impact of circuit limit and device noise on RRAM based conditional generative adversarial network",2020,"China Semiconductor Technology International Conference 2020, CSTIC 2020",,,"9282546","","",,,"10.1109/CSTIC49141.2020.9282546","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85099179307&doi=10.1109%2fCSTIC49141.2020.9282546&partnerID=40&md5=aafd6294130eb4cabc3a064e462a7a7a","Institute of Microelectronics, Peking University, Beijing, 100871, China; Key Laboratory of Microelectronic Devices and Circuits, Peking University, Beijing, 100871, China; Frontiers Science Center for Nano-optoelectronics, Peking University, Beijing, 100871, China","Bao, S., Institute of Microelectronics, Peking University, Beijing, 100871, China; Wang, Z., Institute of Microelectronics, Peking University, Beijing, 100871, China, Key Laboratory of Microelectronic Devices and Circuits, Peking University, Beijing, 100871, China; Liu, T., Institute of Microelectronics, Peking University, Beijing, 100871, China; Chen, D., Institute of Microelectronics, Peking University, Beijing, 100871, China; Cai, Y., Institute of Microelectronics, Peking University, Beijing, 100871, China, Frontiers Science Center for Nano-optoelectronics, Peking University, Beijing, 100871, China; Huang, R., Institute of Microelectronics, Peking University, Beijing, 100871, China, Key Laboratory of Microelectronic Devices and Circuits, Peking University, Beijing, 100871, China","In this work, a Conditional Generative Adversarial Network (CGAN) [1] is demonstrated based on the Resistive Random Access Memory (RRAM). During training, the read noise of RRAM is utilized as a random bias source to enrich the diversity of the generator in CGAN. Further, we evaluate the impact of both read noise (RRAM as weight storage cell) and the resolution of the AD/DA circuit on the performance of CGAN through a comprehensive simulation. © 2020 IEEE.","CGAN; Read Noise; RRAM","Semiconductor device manufacture; Timing circuits; Adversarial networks; Device noise; Read noise; Resistive random access memory (rram); Storage cells; RRAM",,,,,"2018YFB1107701

National Natural Science Foundation of China, NSFC: 61421005, 61834001, 61904003, B18001

China Postdoctoral Science Foundation: 2019M650340","This work was supported in part by the National Key Research and Development Project under grant No. 2018YFB1107701, in part by the National Natural Science Foundation of China under grant No. 61834001, No. 61904003, No. 61421005, and in part by the “111” Project under grant No. B18001. Z. W. acknowledges the support from China Postdoctoral Science Foundation (No. 2019M650340).",,"Mirza, M., (2014); Dai, Z., (2017) Neural Information Processing Systems, NIPS; Wang, Z., (2016) Nanoscale, 8, pp. 14015-14022; Kang, J., (2017) IEEE International Electron Devices Meeting (IEDM)","Wang, Z.; Institute of Microelectronics, China; email: wangzongwei@pku.edu.cn
Cai, Y.; Institute of Microelectronics, China; email: caiyimao@pku.edu.cn","Claeys C.Liang S.Lin Q.Huang R.Wu H.Song P.Lai K.Zhang Y.Zhang B.Qu X.Lung H.-L.Yu W.",,"Institute of Electrical and Electronics Engineers Inc.","2020 China Semiconductor Technology International Conference, CSTIC 2020","26 June 2020 through 17 July 2020",,165992,,9781728165585,,,"English","China Semicond. Technol. Int. Conf. , CSTIC",Conference Paper,"Final","",Scopus,2-s2.0-85099179307
"Rezaei M., Yang H., Meinel C.","57197830197;54788627000;56187776800;","Recurrent generative adversarial network for learning imbalanced medical image semantic segmentation",2020,"Multimedia Tools and Applications","79","21-22",,"15329","15348",,11,"10.1007/s11042-019-7305-1","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85061196523&doi=10.1007%2fs11042-019-7305-1&partnerID=40&md5=de294fddae79715fce454dd2d42cffcd","Hasso Plattner Institute, Prof. Dr. Helmert Street 2-3, Potsdam, Germany","Rezaei, M., Hasso Plattner Institute, Prof. Dr. Helmert Street 2-3, Potsdam, Germany; Yang, H., Hasso Plattner Institute, Prof. Dr. Helmert Street 2-3, Potsdam, Germany; Meinel, C., Hasso Plattner Institute, Prof. Dr. Helmert Street 2-3, Potsdam, Germany","We propose a new recurrent generative adversarial architecture named RNN-GAN to mitigate imbalance data problem in medical image semantic segmentation where the number of pixels belongs to the desired object are significantly lower than those belonging to the background. A model trained with imbalanced data tends to bias towards healthy data which is not desired in clinical applications and predicted outputs by these networks have high precision and low recall. To mitigate imbalanced training data impact, we train RNN-GAN with proposed complementary segmentation mask, in addition, ordinary segmentation masks. The RNN-GAN consists of two components: a generator and a discriminator. The generator is trained on the sequence of medical images to learn corresponding segmentation label map plus proposed complementary label both at a pixel level, while the discriminator is trained to distinguish a segmentation image coming from the ground truth or from the generator network. Both generator and discriminator substituted with bidirectional LSTM units to enhance temporal consistency and get inter and intra-slice representation of the features. We show evidence that the proposed framework is applicable to different types of medical images of varied sizes. In our experiments on ACDC-2017, HVSMR-2016, and LiTS-2017 benchmarks we find consistently improved results, demonstrating the efficacy of our approach. © 2019, Springer Science+Business Media, LLC, part of Springer Nature.","Imbalanced medical image semantic segmentation; Recurrent generative adversarial network","Image segmentation; Long short-term memory; Pixels; Semantic Web; Semantics; Adversarial networks; Clinical application; Image semantics; Imbalance datum; Imbalanced data; Segmentation images; Segmentation masks; Temporal consistency; Medical image processing",,,,,"Universität Potsdam, UP: Hasso Plattner Institute",,,"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Zheng, X., (2015) Tensorflow: Large-Scale Machine Learning on Heterogeneous Systems, , https://www.tensorflow.org/.Softwareavailablefromtensorflow.org; Afshin, M., Ayed, I.B., Punithakumar, K., Law, M., Islam, A., Goela, A., Peters, T., Li, S., Regional assessment of cardiac left ventricular myocardial function via mri statistical features (2014) IEEE Trans Med Imaging, 33 (2), pp. 481-494; Avola, D., Cinque, L., Encephalic nmr image analysis by textural interpretation (2008) Proceedings of the 2008 ACM Symposium on Applied Computing, pp. 1338-1342. , ACM; Avola, D., Cinque, L., Di Girolamo, M., A novel t-cad framework to support medical image analysis and reconstruction (2011) International Conference on Image Analysis and Processing, pp. 414-423. , Springer; Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, P.A., Cetin, I., Ballester, M.A.G., Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: Is the problem solved? (2018) IEEE Transactions on Medical Imaging; Bi, L., Kim, J., Kumar, A., Feng, D., (2017) Automatic Liver Lesion Detection Using Cascaded Deep Residual Networks; Chollet, F., (2015) Keras; Christ, P.F., Ettlinger, F., Grun, F., Elshaer, M.E.A., Lipkova, J., Schlecht, S., Ahmaddy, F., Menze, B.H., (2017) Automatic Liver and Tumor Segmentation of CT and MRI Volumes Using Cascaded Fully Convolutional Neural Networks; Ciecholewski, M., Support vector machine approach to cardiac spect diagnosis (2011) International Workshop on Combinatorial Image Analysis, pp. 432-443. , Springer; Douzas, G., Bacao, F., Effective data generation for imbalanced learning using conditional generative adversarial networks (2018) Expert Syst Appl, 91, pp. 464-471; Drozdzal, M., Chartrand, G., Vorontsov, E., Shakeri, M., Di Jorio, L., Tang, A., Romero, A., Kadoury, S., Learning normalized inputs for iterative estimation in medical image segmentation (2018) Med Image Anal, 44, pp. 1-13; Eslami, A., Karamalis, A., Katouzian, A., Navab, N., Segmentation by retrieval with guided random walks: application to left ventricle segmentation in mri (2013) Med Image Anal, 17 (2), pp. 236-253; Fidon, L., Li, W., Garcia-Peraza-herrera, L.C., Ekanayake, J., Kitchen, N., Ourselin, S., Vercauteren, T., Generalised wasserstein dice score for imbalanced multi-class segmentation using holistic convolutional networks (2017) International MICCAI Brainlesion Workshop, pp. 64-76. , Springer; Fischl, B., Salat, D.H., Van Der Kouwe, A.J., Makris, N., Ségonne, F., Quinn, B.T., Dale, A.M., Sequence-independent segmentation of magnetic resonance images (2004) Neuroimage, 23, pp. S69-S84; Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., (2014) Generative Adversarial Networks ArXiv e-prints; Graves, A., Schmidhuber, J., Framewise phoneme classification with bidirectional lstm and other neural network architectures (2005) Neural Netw, 18 (5-6), pp. 602-610; Han, X., (2017) Automatic Liver Lesion Segmentation Using a Deep Convolutional Neural Network Method; Hashemi, S.R., Salehi, S.S.M., Erdogmus, D., Prabhu, S.P., Warfield, S.K., Gholipour, A., (2018) Tversky as a Loss Function for Highly Unbalanced Image Segmentation Using 3D Fully Convolutional Deep Networks; Inda Maria-Del-Mar, R.B., Seoane, J., Glioblastoma multiforme: A look inside its heterogeneous nature (2014) Cancer Archive, pp. 226-239; Isensee, F., Jaeger, P.F., Full, P.M., Wolf, I., Engelhardt, S., Maier-Hein, K.H., Automatic cardiac disease assessment on cine-mri via time-series segmentation and domain specific features (2017) International Workshop on Statistical Atlases and Computational Models of the Heart, pp. 120-129. , Springer; Ishida, T., Niu, G., Hu, W., Sugiyama, M., Learning from complementary labels (2017) Advances in Neural Information Processing Systems, pp. 5639-5649; Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A., Image-to-image translation with conditional adversarial networks (2017) The IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Jang, J., Eo, T., Kim, M., Choi, N., Han, D., Kim, D., Hwang, D., Medical image matching using variable randomized undersampling probability pattern in data acquisition (2014) 2014 International Conference on Electronics, Information and Communications (ICEIC), pp. 1-2. , https://doi.org/10.1109/ELINFOCOM.2014.6914453; Kaur, R., Juneja, M., Mandal, A., A comprehensive review of denoising techniques for abdominal ct images (2018) Multimedia Tools and Applications, pp. 1-36; Kohl, S., Bonekamp, D., Schlemmer, H., Yaqubi, K., Hohenfellner, M., Hadaschik, B., Radtke, J., Maier-Hein, K.H., (2017) Adversarial Networks for the Detection of Aggressive Prostate Cancer; LeCun, Y., Bengio, Y., Hinton, G., Deep learning (2015) Nature, 521 (7553), pp. 436-444; Mahapatra, D., Automatic cardiac segmentation using semantic information from random forests (2014) J Digit Imaging, 27 (6), pp. 794-804; Mirza, M., Osindero, S., (2014) Conditional Generative Adversarial Nets.; Moeskops, P., Veta, M., Lafarge, M.W., Eppenhof, K.A.J., Pluim, J.P.W., (2017) Adversarial Training and Dilated Convolutions for Brain MRI Segmentation.; Nasr, G.E., Badr, E., Joun, C., Cross entropy error function in neural networks: Forecasting gasoline demand (2002) FLAIRS Conference, pp. 381-384; Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A., Context encoders: Feature learning by inpainting (2016) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536-2544; Peng, P., Lekadir, K., Gooya, A., Shao, L., Petersen, S.E., Frangi, A.F., A review of heart chamber segmentation for structural and functional analysis using cardiac magnetic resonance imaging (2016) Magn Reson Mater Phys, Biol Med, 29 (2), pp. 155-195; Pohl, K.M., Fisher, J., Grimson, W.E.L., Kikinis, R., Wells, W.M., A bayesian model for joint segmentation and registration (2006) Neuroimage, 31 (1), pp. 228-239; Poudel, R.P., Lamata, P., Montana, G., Recurrent fully convolutional neural networks for multi-slice mri cardiac segmentation (2016) In: Reconstruction, Segmentation, and Analysis of Medical Images, pp. 83-94. , Springer; Prabhu, V., Kuppusamy, P., Karthikeyan, A., Varatharajan, R., Evaluation and analysis of data driven in expectation maximization segmentation through various initialization techniques in medical images (2018) Multimed Tools Appl, 77 (8), pp. 10375-10390; Qiu, Q., Song, Z., A nonuniform weighted loss function for imbalanced image classification (2018) Proceedings of the 2018 International Conference on Image and Graphics Processing, pp. 78-82. , ACM; Rohé, M.M., Sermesant, M., Pennec, X., Automatic multi-atlas segmentation of myocardium with svf-net (2017) Statistical Atlases and Computational Modeling of the Heart (STACOM) Workshop; Ronneberger, O., Fischer, P., Brox, T., U-net: Convolutional networks for biomedical image segmentation (2015) International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234-241. , Springer International Publishing; Rota Bulo, S., Neuhold, G., Kontschieder, P., Loss max-pooling for semantic image segmentation (2017) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2126-2135; Shahzad, R., Gao, S., Tao, Q., Dzyubachyk, O., van Der Geest, R., Automated cardiovascular segmentation in patients with congenital heart disease from 3d cmr scans: Combining multi-atlases and level-sets (2016) Reconstruction, Segmentation, and Analysis of Medical Images, pp. 147-155; Sudre, C.H., Li, W., Vercauteren, T., Ourselin, S., Cardoso, M.J., Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations (2017) Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 240-248. , Springer; Tustison, N.J., Avants, B.B., Cook, P.A., Zheng, Y., Egan, A., Yushkevich, P.A., Gee, J.C., N4itk: improved n3 bias correction (2010) IEEE Trans Med Imaging, 29 (6), pp. 1310-1320; Vorontsov, E., Tang, A., Pal, C., Kadoury, S., Liver lesion segmentation informed by joint liver segmentation (2018) 15Th IEEE International Symposium on Biomedical Imaging, pp. 1332-1335. , ISBI 2018); Vorontsov, E., Tang, A., Pal, C., Kadoury, S., Liver lesion segmentation informed by joint liver segmentation (2018) 15Th IEEE International Symposium on Biomedical Imaging (ISBI 2018), pp. 1332-1335; Wolterink, J.M., Leiner, T., Viergever, M.A., Išgum, I., Dilated convolutional neural networks for cardiovascular mr segmentation in congenital heart disease (2016) Reconstruction, Segmentation, and Analysis of Medical Images, pp. 95-102. , Springer; Wolterink, J.M., Leiner, T., Viergever, M.A., Isgum, I., (2017) Automatic Segmentation and Disease Classification Using Cardiac Cine Mr Images.; Xu, J., Schwing, A.G., Urtasun, R., Tell me what you see and i will show you where it is (2014) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3190-3197; Xue, Y., Xu, T., Zhang, H., Long, L.R., Huang, X., (2017) Segan: Adversarial Network with Multi-Scalel1 Loss for Medical Image Segmentation.; Yu, L., Yang, X., Qin, J., Heng, P.A., 3d fractalnet: Dense volumetric segmentation for cardiovascular mri volumes (2016) Reconstruction, Segmentation, and Analysis of Medical Images, pp. 103-110. , Springer; Yu, X., Liu, T., Gong, M., Tao, D., Learning with biased complementary labels (2018) The European Conference on Computer Vision (ECCV); Zhang, Y.D., Muhammad, K., Tang, C., Twelve-layer deep convolutional neural network with stochastic pooling for tea category classification on gpu platform (2018) Multimedia Tools and Applications, pp. 1-19; Zhang, Y.D., Zhao, G., Sun, J., Wu, X., Wang, Z.H., Liu, H.M., Govindaraj, V.V., Li, J., Smart pathological brain detection by synthetic minority oversampling technique, extreme learning machine, and jaya algorithm (2017) Multimedia Tools and Applications, pp. 1-20; Zhou, Y., Berg, T.L., Learning temporal transformations from time-lapse videos (2016) European Conference on Computer Vision, pp. 262-277; Zhu, J.Y., Park, T., Isola, P., Efros, A.A., Unpaired image-to-image translation using cycle-consistent adversarial networks (2017) The IEEE International Conference on Computer Vision (ICCV); Zhu, W., Xie, X., (2016) Adversarial Deep Structural Networks for Mammographic Mass Segmentation; Zotti, C., Luo, Z., Humbert, O., Lalande, A., Jodoin, P.M., (2017) Gridnet with Automatic Shape Prior Registration for Automatic Mri Cardiac Segmentation","Rezaei, M.; Hasso Plattner Institute, Prof. Dr. Helmert Street 2-3, Germany; email: mina.rezaei@hpi.de",,,"Springer",,,,,13807501,,MTAPF,,"English","Multimedia Tools Appl",Article,"Final","",Scopus,2-s2.0-85061196523
"Roy D., Mukherjee D., Chanda B.","57219732152;7202334792;7007109395;","An unsupervised approach towards varying human skin tone using generative adversarial networks",2020,"Proceedings - International Conference on Pattern Recognition",,,"9412852","10681","10688",,,"10.1109/ICPR48806.2021.9412852","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85110534503&doi=10.1109%2fICPR48806.2021.9412852&partnerID=40&md5=86dc8b33ceba5035030cf4e82617a330","Indian Statistical Institute, Kolkata, India","Roy, D., Indian Statistical Institute, Kolkata, India; Mukherjee, D., Indian Statistical Institute, Kolkata, India; Chanda, B., Indian Statistical Institute, Kolkata, India","With the increasing popularity of augmented and virtual reality, retailers are now focusing more towards customer satisfaction to increase the amount of sales. Although augmented reality is not a new concept but it has gained much needed attention over the past few years. Our present work is targeted towards this direction which may be used to enhance user experience in various virtual and augmented reality based applications. We propose a model to change skin tone of a person. Given any input image of a person or a group of persons with some value indicating the desired change of skin color towards fairness or darkness, this method can change the skin tone of the persons in the image. This is an unsupervised method and also unconstrained in terms of pose, illumination, number of persons in the image etc. The goal of this work is to reduce the time and effort which is generally required for changing the skin tone using existing applications (e.g., Photoshop) by professionals or novice. To establish the efficacy of this method we have compared our result with that of some popular photo editor and also with the result of some existing benchmark method related to human attribute manipulation. Rigorous experiments on different datasets show the effectiveness of this method in terms of synthesizing perceptually convincing outputs. © 2020 IEEE",,"Augmented reality; Sales; User experience; Adversarial networks; Augmented and virtual realities; Human attributes; Input image; Photo editors; Unsupervised approaches; Unsupervised method; Virtual and augmented reality; Pattern recognition",,,,,,,,"https://en.wikipedia.org/wiki/Augmented_reality; https://www.technologyreview.com/2019/10/23/238473/augmentedreality-in-retail-virtual-try-before-you-buy/; https://retailwire.com/discussion/; https://p4t6u7k5.stackpathcdn.com/wp-content/uploads/shutterstock605616227.jpg; https://designpanoply.com/blog/how-to-change-a-persons-skin-color-from-dark-to-light-in-photoshop; https://www.youtube.com/watch?v=pO7gq_2BvZw; https://i1.wp.com/cbtpsychology.com/wp-content/uploads/2018/10/Therapy.jpg; Al-Mohair, H.K., Saleh, J.M., Suandi, S.A., Hybrid human skin detection using neural network and k-means clustering technique (2015) Applied Soft Computing, 33, pp. 337-347; Brand, J., Mason, J.S., A comparative assessment of three approaches to pixel-level human skin-detection (2000) Proceedings 15th International Conference on Pattern Recognition. ICPR-2000, 1, pp. 1056-1059; Chakravarti, L., (1967) Roy, Handbook of Methods of Applied Statistics, p. 1; Chen, C.-W., Huang, D.-Y., Fuh, C.-S., Automatic skin color beautification (2009) International Conference on Arts and Technology, pp. 157-164. , Springer; Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L., ImageNet: A large-scale hierarchical image database (2009) CVPR09; Dong, H., Liang, X., Shen, X., Wang, B., Lai, H., Zhu, J., Hu, Z., Yin, J., Towards multi-pose guided virtual try-on network (2019) Proceedings of the IEEE International Conference on Computer Vision, pp. 9026-9035; Gong, K., Liang, X., Zhang, D., Shen, X., Lin, L., Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing (2017) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 932-940; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Advances in Neural Information Processing Systems, pp. 2672-2680; He, Z., Zuo, W., Kan, M., Shan, S., Chen, X., Attgan: Facial attribute editing by only changing what you want (2019) IEEE Transactions on Image Processing, 28 (11), pp. 5464-5478; Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S., Gans trained by a two time-scale update rule converge to a local nash equilibrium (2017) Advances in Neural Information Processing Systems, pp. 6626-6637; Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A., Image-to-image translation with conditional adversarial networks (2017) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125-1134; Jo, Y., Park, J., Sc-fegan: Face editing generative adversarial network with user's sketch and color (2019) Proceedings of the IEEE International Conference on Computer Vision, pp. 1745-1753; Johnson, J., Alahi, A., Fei-Fei, L., Perceptual losses for real-time style transfer and super-resolution (2016) European Conference on Computer Vision, pp. 694-711. , Springer; Kakumanu, P., Makrogiannis, S., Bourbakis, N., A survey of skin-color modeling and detection methods (2007) Pattern Recognition, 40 (3), pp. 1106-1122; Kanzawa, Y., Kimura, Y., Naito, T., Human skin detection by visible and near-infrared imaging (2011) IAPR Conference on Machine Vision Applications, 12, pp. 14-22. , Citeseer; Kolkur, S., Kalbande, D., Shimpi, P., Bapat, C., Jatakia, J., (2017) Human Skin Detection Using Rgb, Hsv and Ycbcr Color Models, , arXiv preprint; Liu, Z., Luo, P., Qiu, S., Wang, X., Tang, X., DeepFashion: Powering robust clothes recognition and retrieval with rich annotations (2016) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1096-1104; Mirza, M., Osindero, S., (2014) Conditional Generative Adversarial Nets, , arXiv preprint; Naji, S., Jalab, H.A., Kareem, S.A., A survey on skin detection in colored images (2019) Artificial Intelligence Review, 52 (2), pp. 1041-1087; Newell, A., Yang, K., Deng, J., Stacked hourglass networks for human pose estimation (2016) European Conference on Computer Vision, pp. 483-499. , Springer; Nguyen-Trang, T., A new efficient approach to detect skin in color image using Bayesian classifier and connected component algorithm (2018) Mathematical Problems in Engineering, p. 2018; Nunez, A.S., Mendenhall, M.J., Detection of human skin in near infrared hyperspectral imagery (2008) IGARSS 2008-2008 IEEE International Geoscience and Remote Sensing Symposium, 2, pp. II-621; Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X., Improved techniques for training gans (2016) Advances in Neural Information Processing Systems, pp. 2234-2242; Shaik, K.B., Ganesan, P., Kalist, V., Sathish, B.S., Jenitha, J.M.M., Comparative study of skin color detection and segmentation in hsv and ycbcr color space (2015) Procedia Computer Science, 57 (12), pp. 41-48; Shen, W., Liu, R., Learning residual images for face attribute manipulation (2017) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4030-4038; Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., Rethinking the inception architecture for computer vision (2016) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818-2826; Tan, W.R., Chan, C.S., Yogarajah, P., Condell, J., A fusion approach for efficient human skin detection (2011) IEEE Transactions on Industrial Informatics, 8 (1), pp. 138-147; Wang, Y., Wang, S., Qi, G., Tang, J., Li, B., Weakly supervised facial attribute manipulation via deep adversarial network (2018) 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 112-121; Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P., Image quality assessment: From error visibility to structural similarity (2004) IEEE Transactions on Image Processing, 13 (4), pp. 600-612; Wang, Z., Simoncelli, E.P., Bovik, A.C., Multiscale structural similarity for image quality assessment (2003) The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2, pp. 1398-1402. , Ieee, 2003; Zhang, J., Shu, Y., Xu, S., Cao, G., Zhong, F., Liu, M., Qin, X., Sparsely grouped multi-task generative adversarial networks for facial attribute manipulation (2018) Proceedings of the 26th ACM International Conference on Multimedia, pp. 392-401; Zuo, H., Fan, H., Blasch, E., Ling, H., Combining convolutional and recurrent neural networks for human skin detection (2017) IEEE Signal Processing Letters, 24 (3), pp. 289-293",,,,"Institute of Electrical and Electronics Engineers Inc.","25th International Conference on Pattern Recognition, ICPR 2020","10 January 2021 through 15 January 2021",,169954,10514651,9781728188089,PICRE,,"English","Proc. Int. Conf. Pattern Recognit.",Conference Paper,"Final","All Open Access, Green",Scopus,2-s2.0-85110534503
"Ezeme O.M., Mahmoud Q.H., Azim A.","57215424037;6701548037;36023296200;","Design and development of AD-CGAN: Conditional generative adversarial networks for anomaly detection",2020,"IEEE Access","8",,,"177667","177681",,,"10.1109/ACCESS.2020.3025530","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85102837802&doi=10.1109%2fACCESS.2020.3025530&partnerID=40&md5=62cc8ac82cb909a4f4d3f6d15b5a9b79","Department of Electrical, Computer, and Software Engineering, Ontario Tech University, Oshawa, ON  L1G 0C5, Canada","Ezeme, O.M., Department of Electrical, Computer, and Software Engineering, Ontario Tech University, Oshawa, ON  L1G 0C5, Canada; Mahmoud, Q.H., Department of Electrical, Computer, and Software Engineering, Ontario Tech University, Oshawa, ON  L1G 0C5, Canada; Azim, A., Department of Electrical, Computer, and Software Engineering, Ontario Tech University, Oshawa, ON  L1G 0C5, Canada","Whether in the realm of software or hardware, datasets representing the state of systems are mostly imbalanced. This imbalance is because these systems' reliability requirements make the occurrence of an anomaly a rare phenomenon. Hence, most datasets on anomaly detection have a relatively small percentage that captures the anomaly. Recently, generative adversarial networks (GAN) have shown promising results in image generation tasks. Therefore, in this research work, we build on conditional GANs (CGAN) to generate plausible distributions of a given profile to solve the challenge of data imbalance in anomaly detection tasks and present a novel framework for anomaly detection. Firstly, we learn the pattern of the minority class data samples using a single class CGAN. Secondly, we use the knowledge base of the single class CGAN to generate samples that augment the minority class samples so that a binary class CGAN can train on the typical and malicious profiles with a balanced dataset. This approach inherently eliminates the bias imposed on algorithms from the dataset and results in a robust framework with improved generalization. Thirdly, the binary class CGAN generates a knowledge base that we use to construct the cluster-based anomaly detector. During testing, we do not use the single class CGAN, thereby providing us with a lean and efficient algorithm for anomaly detection that can do anomaly detection on semi-supervised and non-parametric multivariate data.We test the framework on logs and image-based anomaly detection datasets with class imbalance.We compare the performance of AD-CGAN with GAN-derived and non-GAN-derived state of the art algorithms on benchmark datasets. AD-CGAN outperforms most of the algorithms in the standard metrics of Precision, Recall, and F-1 Score. Where AD-CGAN does not perform better in the parameters used, it has the advantage of being lightweight. Therefore, it can be deployed for both online and offline anomaly detection tasks since it does not use an input sample inversion strategy. © 2020 Institute of Electrical and Electronics Engineers Inc.. All rights reserved.","Anomaly detection; Deep learning; Generative adversarial networks; Transfer learning","Benchmarking; Knowledge based systems; Adversarial networks; Anomaly detector; Benchmark datasets; Design and Development; Image generations; Reliability requirements; Standard metrics; State-of-the-art algorithms; Anomaly detection",,,,,,,,"Wang, H., Bah, M.J., Hammad, M., Progress in outlier detection techniques: A survey (2019) IEEE Access, 7, pp. 107964-108000; Luo, Y., Xiao, Y., Cheng, L., Peng, G., Yao, D., Deep learning-based anomaly detection in cyber-physical systems: Progress and opportunities (2020) ACM Comput. Surv., p. 29. , Mar; Hawkins, D.M., (1980) Identification of Outliers, 11. , Cham, Switzerland: Springer; Liu, F.T., Ting, K.M., Zhou, Z., Isolation-based anomaly detection (2012) ACM Trans. Knowl. Discovery Data, 6 (1), pp. 1-39. , Mar; (2017) A. C. Bahnsen, , https://blog.easysol.net/building-ai-applications/, Jun; Chalapathy, R., Chawla, S., (2019) Deep Learning for Anomaly Detection: A Survey, , http://arxiv.org/abs/1901.03407; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Proc. Adv. Neural Inf. Process. Syst., pp. 2672-2680; Wang, Z., She, Q., Ward, T.E., (2019) Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy, , http://arxiv.org/abs/1906.01529; Mirza, M., Osindero, S., (2014) Conditional Generative Adversarial Nets, , http://arxiv.org/abs/1411.1784; Zhang, G.P., Time series forecasting using a hybrid ARIMA and neural network model (2003) Neurocomputing, 50, pp. 159-175. , Jan; Chandola, V., Banerjee, A., Kumar, V., Anomaly detection: A survey (2009) ACM Comput. Surv., 41 (3), p. 15; Mutz, D., Valeur, F., Vigna, G., Kruegel, C., Anomalous system call detection (2006) ACM Trans. Inf. Syst. Secur., 9 (1), pp. 61-93. , Feb; Ezeme, M., Azim, A., Mahmoud, Q.H., An imputation-based augmented anomaly detection from large traces of operating system events (2017) Proc. 4th, IEEE/ACM Int. Conf. Big Data Comput., Appl. Technol. (BDCAT), pp. 43-52; Ezeme, O.M., Azim, A., Mahmoud, Q., PESKEA: Anomaly detection framework for profiling kernel event attributes in embedded systems (2020) IEEE Trans. Emerg. Topics Comput., , Early access, Feb. 3; Du, M., Li, F., Zheng, G., Srikumar, V., DeepLog: Anomaly detection and diagnosis from system logs through deep learning (2017) Proc. ACM SIGSAC Conf. Comput. Commun. Secur., pp. 1285-1298. , Oct; Xu, W., Huang, L., Fox, A., Patterson, D., Jordan, M., Largescale system problem detection by mining console logs (2009) Proc. SOSP; Ezeme, M.O., Mahmoud, Q.H., Azim, A., Hierarchical attentionbased anomaly detection model for embedded operating systems (2018) Proc. IEEE 24th Int. Conf. Embedded Real-Time Comput. Syst. Appl. (RTCSA), pp. 225-231. , Aug; Ezeme, O.M., Mahmoud, Q.H., Azim, A., DReAM: Deep recursive attentive model for anomaly detection in kernel events (2019) IEEE Access, 7, pp. 18860-18870; Yoon, M.-K., Mohan, S., Choi, J., Christodorescu, M., Sha, L., Learning execution contexts from system call distribution for anomaly detection in smart embedded system (2017) Proc. 2nd Int. Conf. Internet-of-Things Design Implement., pp. 191-196. , Apr; Gu, Y., McCallum, A., Towsley, D., Detecting anomalies in network traffic using maximum entropy estimation (2005) Proc. 5th ACM SIGCOMM Conf. Internet Meas. (IMC), p. 32; Salem, M., Crowley, M., Fischmeister, S., Anomaly detection using inter-arrival curves for real-time systems (2016) Proc. 28th Euromicro Conf. Real-Time Syst. (ECRTS), pp. 97-106. , Jul; Li, F., Li, Z., Huo, W., Feng, X., Locating software faults based on minimum debugging frontier set (2017) IEEE Trans. Softw. Eng., 43 (8), pp. 760-776. , Aug; Kosoresow, A.P., Hofmeyer, S.A., Intrusion detection via system call traces (1997) IEEE Softw., 14 (5), pp. 35-42. , Sep; Warrender, C., Forrest, S., Pearlmutter, B., Detecting intrusions using system calls: Alternative data models (1999) Proc. IEEE Symp. Secur. Privacy, pp. 133-145. , May; Hofmeyr, S.A., Forrest, S., Somayaji, A., Intrusion detection using sequences of system calls (1998) J. Comput. Secur., 6 (3), pp. 151-180. , Jul; Ezeme, O.M., Lescisin, M., Mahmoud, Q.H., Azim, A., Deepanom: An ensemble deep framework for anomaly detection in system processes (2019) Proc. Can. Conf. Artif. Intell., pp. 549-555. , Cham, Switzerland: Springer; Ezeme, O.M., Mahmoud, Q., Azim, A., A framework for anomaly detection in time-driven and event-driven processes using kernel traces (2020) IEEE Trans. Knowl. Data Eng., , Early access, Mar. 5; Han, T., Liu, C., Yang, W., Jiang, D., Deep transfer network with joint distribution adaptation: A new intelligent fault diagnosis framework for industry application (2020) ISA Trans., 97, pp. 269-281. , Feb; Li, D., Chen, D., Jin, B., Shi, L., Goh, J., Ng, S.-K., MAD-GAN: Multivariate anomaly detection for time series data with generative adversarial networks (2019) Proc. Int. Conf. Artif. Neural Netw., pp. 703-716. , Cham, Switzerland: Springer; Schlegl, T., Seeböck, P., Waldstein, S.M., Ursula, S.-E., Georg, L., Unsupervised anomaly detection with generative adversarial networks to guide marker discovery (2017) Proc. Int. Conf. Inf. Process. Med. Imag., pp. 146-157. , Cham, Switzerland: Springer; Zenati, H., Sheng Foo, C., Lecouat, B., Manek, G., Chandrasekhar, V.R., (2018) Efficient GAN-based Anomaly Detection, , http://arxiv.org/abs/1802.06222; Han, T., Liu, C., Yang, W., Jiang, D., A novel adversarial learning framework in deep convolutional neural network for intelligent diagnosis of mechanical faults (2019) Knowl.-Based Syst., 165, pp. 474-487. , Feb; Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P., Infogan: Interpretable representation learning by information maximizing generative adversarial nets (2016) Proc. Adv. Neural Inf. Process. Syst., pp. 2172-2180; Hinton, G.E., Roweis, S.T., Stochastic neighbor embedding (2003) Proc. Adv. Neural Inf. Process. Syst., pp. 857-864; Zou, H., Hastie, T., Tibshirani, R., Sparse principal component analysis (2006) J. Comput. Graph. Statist., 15 (2), pp. 265-286. , Jun; Goldstein, M., Uchida, S., A comparative evaluation of unsupervised anomaly detection algorithms for multivariate data (2016) PLoS ONE, 11 (4). , Apr; Wattenberg, M., Viégas, F., Johnson, I., How to use T-SNE effectively (2016) Distill, 1 (10), p. e2. , http://distill.pub/2016/misread-tsne; Campos, G.O., Zimek, A., Sander, J., Campello, R.J.G.B., Micenková, B., Schubert, E., Assent, I., Houle, M.E., On the evaluation of unsupervised outlier detection: Measures, datasets, and an empirical study (2016) Data Mining Knowl. Discovery, 30 (4), pp. 891-927. , Jul; Zhai, S., Cheng, Y., Lu, W., Zhang, Z., (2016) Deep Structured Energy Based Models for Anomaly Detection, , http://arxiv.org/abs/1605.07717","Ezeme, O.M.; Department of Electrical, Canada; email: mellitus.ezeme@ontariotechu.net",,,"Institute of Electrical and Electronics Engineers Inc.",,,,,21693536,,,,"English","IEEE Access",Article,"Final","All Open Access, Gold",Scopus,2-s2.0-85102837802
"Shashank A., Sajithvariyar V.V., Sowmya V., Soman K.P., Sivanpillai R., Brown G.K.","57220023850;57189307235;36096164300;57205365723;8970929800;7406467671;","Identifying epiphytes in drones photos with a conditional generative adversarial network (C-GAN)",2020,"International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives","44","M-2",,"99","104",,2,,"https://www.scopus.com/inward/record.uri?eid=2-s2.0-85097566749&partnerID=40&md5=6edfcde607c994f9b47f5ecaa40c2cb3","Center for Computational Engineering and Networking, Amrita School of Engineering, Amrita Vishwa Vidyapeetham, Coimbatore, TN  641 112, India; Wyoming GIS Center, University of Wyoming, Laramie, WY  82072, United States; Department of Botany, University of Wyoming, Laramie, WY  82072, United States","Shashank, A., Center for Computational Engineering and Networking, Amrita School of Engineering, Amrita Vishwa Vidyapeetham, Coimbatore, TN  641 112, India; Sajithvariyar, V.V., Center for Computational Engineering and Networking, Amrita School of Engineering, Amrita Vishwa Vidyapeetham, Coimbatore, TN  641 112, India; Sowmya, V., Center for Computational Engineering and Networking, Amrita School of Engineering, Amrita Vishwa Vidyapeetham, Coimbatore, TN  641 112, India; Soman, K.P., Center for Computational Engineering and Networking, Amrita School of Engineering, Amrita Vishwa Vidyapeetham, Coimbatore, TN  641 112, India; Sivanpillai, R., Wyoming GIS Center, University of Wyoming, Laramie, WY  82072, United States; Brown, G.K., Department of Botany, University of Wyoming, Laramie, WY  82072, United States","Unmanned Aerial Vehicle (UAV) missions often collect large volumes of imagery data. However, not all images will have useful information, or be of sufficient quality. Manually sorting these images and selecting useful data are both time consuming and prone to interpreter bias. Deep neural network algorithms are capable of processing large image datasets and can be trained to identify specific targets. Generative Adversarial Networks (GANs) consist of two competing networks, Generator and Discriminator that can analyze, capture, and copy the variations within a given dataset. In this study, we selected a variant of GAN called Conditional-GAN that incorporates an additional label parameter, for identifying epiphytes in photos acquired by a UAV in forests within Costa Rica. We trained the network with 70%, 80%, and 90% of 119 photos containing the target epiphyte, Werauhia kupperiana (Bromeliaceae) and validated the algorithm's performance using a validation data that were not used for training. The accuracy of the output was measured using structural similarity index measure (SSIM) index and histogram correlation (HC) coefficient. Results obtained in this study indicated that the output images generated by C-GAN were similar (average SSIM = 0.89 - 0.91 and average HC 0.97 - 0.99) to the analyst annotated images. However, C-GAN had difficulty to identify when the target plant was away from the camera, was not well lit, or covered by other plants. Results obtained in this study demonstrate the potential of C-GAN to reduce the time spent by botanists to identity epiphytes in images acquired by UAVs. © 2020 International Society for Photogrammetry and Remote Sensing. All rights reserved.","CNN; Image translation; PIX2PIX; Segmentation; UAV; UNET","Antennas; Deep neural networks; Drones; Large dataset; Remote sensing; Adversarial networks; Algorithm's performance; Bromeliaceae; Large images; Large volumes; Neural network algorithm; Structural similarity index measures (SSIM); Validation data; Image acquisition",,,,,,,,"Acevedo, M. A., Beaudrot, L., Melendez-Ackerman, E., Tremblay, R., Local extinction risk under climate change in a neotropical asymmetrically dispersed epiphyte (2020) Journal of Ecology, 108, pp. 1553-1564. , doi.org; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative Adversarial nets (2014) Proceedings of the Neural Information Processing Systems, , 2014: doi.org/papers.nips.cc/paper/5423-generative-adversarial-nets; Isola, P., Zhu, JY., Zhou, T., Efros, A.A., Image-to-image translation with conditional adversarial networks (2017) IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967-5976. , doi.org; Lou, Z., Ding, S., Object detection in remote sensing images based on GAN (2019) International Conference on Artificial Intelligence and Computer Science (AICS 2019), pp. 499-503. , doi.org; Ma, J., Zhou, F., Multi-poses face frontalization based on pose weighted GAN (2019) IIEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), pp. 1271-1276. , doi.org; Qaio, T., Zhang, J., Xu, D., Tao, D., MirrorGAN: Learning text-to-image generation by redescription (2019) IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1505-1514. , doi.org; Ronneberger, O., Fischer, P., Brox, T., U-net: Convolutional networks for biomedical image segmentation (2015) International Conference on Medical image computing and computer-assisted intervention (MICCAI-2015), pp. 234-241. , doi.org; Russell, B.C., Torralbe, A., Murphy, K.P., Freeman, W.T., LabelMe: A database and web-based tool for image annotation (2008) International Journal of Computer vision, 77, pp. 157-173. , doi.org; Singh, V., Misra, A.K., Detection of plant leaf diseases using image segmentation and soft computing techniques Information Processing in Agriculture, 4 (1), pp. 41-49. , doi.org; Sivanpillai, R., Brown, G.K., Gellis, B.S., Flying UAVs in constrained environments: Best practices for flying within complex forest canopies (2019) Applications of small unmanned aircraft systems: best practices and case studies, pp. 269-282. , Ed. Sharma, J.B., CRC Press, New York, NY; Sun, Y., Liu, Y., Wang., G., Zhang, H., Deep Learning for Plant Identification in Natural Environment (2017) Computational Intelligence and Neurosciences, p. 7361042. , Article ID doi.org; Sun, Y., Yu, W., Chen, Y., Kadam, A., Time series anomaly detection based on GAN (2019) Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS), pp. 375-382. , doi.org; Swain, M.J., Ballard, D.H., Color indexing (1991) International journal of computer vision, 7 (1), pp. 11-22. , doi.org; Wang, Z., Bovik, A. C., Sheikh, H. R., Simoncelli, E. P., Image quality assessment: from error visibility to structural similarity (2004) IEEE transactions on image processing, 13 (4), pp. 600-612. , doi.org","Sajithvariyar, V.V.; Center for Computational Engineering and Networking, India; email: vv_sajithvariyar@cb.amrita.edu","Schuckman K.Jordan T.",,"International Society for Photogrammetry and Remote Sensing","American Society for Photogrammetry and Remote Sensing, ASPRS 2020 Annual Conference","22 June 2020 through 26 June 2020",,165415,16821750,,,,"English","Int. Arch. Photogramm., Remote Sens. Spat. Inf. Sci. - ISPRS Arch.",Conference Paper,"Final","",Scopus,2-s2.0-85097566749
"Li A., Fang J., Jiang Q., Zhou B., Jia Y.","57217320818;57215342708;57215350135;55501961000;34770316700;","A Graph Data Privacy-Preserving Method Based on Generative Adversarial Networks",2020,"Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)","12343 LNCS",,,"227","239",,1,"10.1007/978-3-030-62008-0_16","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85096577205&doi=10.1007%2f978-3-030-62008-0_16&partnerID=40&md5=948317011c0e36dc2090207107345ee5","School of Computer Science, National University of Defense Technology, Changsha, China","Li, A., School of Computer Science, National University of Defense Technology, Changsha, China; Fang, J., School of Computer Science, National University of Defense Technology, Changsha, China; Jiang, Q., School of Computer Science, National University of Defense Technology, Changsha, China; Zhou, B., School of Computer Science, National University of Defense Technology, Changsha, China; Jia, Y., School of Computer Science, National University of Defense Technology, Changsha, China","We proposed a graph anonymization method which is based on a feature learning model of Generative Adversarial Network (GAN). We used the differential privacy to ensure the privacy and take both anonymity and utility into consideration. The method consists of the following two parts: Firstly, we designed a graph feature learning method based on GAN. The method used the bias random walk strategy to sample the node sequence from graph data, and trained the GAN model. After training, the GAN generated a set of simulation sequences that are highly like the real sampled sequence. Secondly, we proposed an anonymous graph construction method based on the simulation node sequence. We calculated the number of edges in the node sequences and constructed a probability adjacency matrix. The differential privacy noise is added to get the anonymous probability adjacency matrix. Then we extract the edges from the anonymous matrix and then constructed the anonymous graph. We evaluate our methodology, showing that the model had good feature learning ability through embedding visualization and link prediction experiments, compared with other anonymous graphs. Through experiments such as metric evaluation, community detection, and de-anonymization attack, we proved that the anonymous method we proposed is better than the current mainstream anonymous method. © 2020, Springer Nature Switzerland AG.","Differential privacy; Generative Adversarial Networks; Graph data; Privacy preservation","Graph structures; Information systems; Information use; Machine learning; Matrix algebra; Privacy by design; Systems engineering; Adjacency matrices; Adversarial networks; Community detection; Differential privacies; Feature learning; Metric evaluation; Random walk strategies; Sampled sequences; Graph theory",,,,,"2019B010136003

National Natural Science Foundation of China, NSFC: 61672020, 61732004, 61732022

National Key Research and Development Program of China, NKRDPC: 2016QY03D0601, 2016QY03D0603, 2017YFB0802204, 2017YFB0803301, 2019QY1406","Acknowledgements. The work described in this paper is partially supported by the National Key Research and Development Program of China (No. 2017YFB0802204, 2016QY03D0603, 2016QY03D0601, 2017YFB0803301, 2019QY1406), the Key R&D Program of Guangdong Province (No. 2019B010136003), and the National Natural Science Foundation of China (No. 61732004, 61732022, 61672020).",,"Ji, S., Li, W., Mittal, P., Secgraph: A uniform and open-source evaluation system for graph data anonymization and de-anonymization (2015) 24Th USENIX Security Symposium USENIX Security 15), pp. 303-318; Sweeney, L., K-anonymous: A model for protecting privacy (2002) Int. J. Uncertainty, Fuzziness Knowl. Based Syst., 10 (5), pp. 557-570; Sun, X., Wang, H., Li, J., Zhang, Y., Satisfying privacy requirements before data anonymization (2012) Comput. J., 55 (4), pp. 422-437; Li, N., Li, T., Venkatasubramanian, S., T-closeness: Privacy beyond k-anonymous and l-diversity (2007) 2007 IEEE 23Rd International Conference on Data Engineering, Pp. 106–115. IEEE; Dwork, C., Differential privacy (2011) Encyclopedia of Cryptography and Security, pp. 338-340; Liu, C., Mittal, P., LinkMirage: Enabling privacy-preserving analytics on social relationships (2016) NDSS 2016; Bhagat, S., Cormode, G., Krishnamurthy, B., Class-based graph anonymization for social network data (2009) Proc. VLDB Endowment, 2 (1), pp. 766-777; Yi, X., Zhang, Y., Privacy-preserving distributed association rule mining via semi-trusted mixer (2007) Data Knowl. Eng., 63 (2), pp. 550-567; Rossi, L., Musolesi, M., Torsello, A., On the k-anonymization of time-varying and multi-layer social graphs (2015) Ninth International AAAI Conference on Web and Social Media; Li, M., Sun, X., Wang, H., Privacy-aware access control with trust management in web service (2011) World Wide Web-Internet Web Inf. Syst., 14 (4), pp. 407-430; Wang, H., Zhang, Z., Taleb, T., Editorial: Special issue on security and privacy of IoT (2017) World Wide Web-Internet & Web Information Systems, pp. 1-6; Sala, A., Zhao, X., Wilson, C., Sharing graphs using differentially private graph models (2011) Proceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference, pp. 81-98. , ACM; Gao, T., Li, F., Sharing social networks using a novel differentially private graph model (2019) 2019 16Th IEEE Annual Consumer Communications & Networking Conference (CCNC), pp. 1-4; Jorgensen, Z., Yu, T., Cormode, G., Publishing attributed social graphs with formal privacy guarantees (2016) The 2016 International Conference, , ACM; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Generative adversarial nets (2014) Advances in Neural Information Processing Systems, pp. 2672-2680. , In; Kipf, T.N., Welling, M., (2016) Semi-Supervised Classification with Graph Convolutional Networks., , arXiv preprint arXiv; Wang, H., Wang, J., Wang, J., Graphgan: Graph representation learning with generative adversarial nets (2018) Thirty-Second AAAI Conference on Artificial Intelligence; Grover, A., Leskovec, J., Node2vec: Scalable feature learning for networks (2016) Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 855-864. , ACM; Sundermeyer, M., Schlüter, R., Ney, H., LSTM neural networks for language modeling (2012) Interspeech, pp. 601-608; Arjovsky, M., Chintala, S., Bottou, L., Wasserstein generative adversarial networks (2017) International Conference on Machine Learning, pp. 214-223; Blondel, V.D., Guillaume, J.-L., Lambiotte, R., Fast unfolding of communities in large networks (2008) J. Stat. Mech. Theory Exp, 2008 (10), p. P10008; Ji, S., Gu, Q., Weng, H., De-Health: All Your Online Health Information are Belong to Us","Li, A.; School of Computer Science, China; email: liaiping@nudt.edu.cn","Huang Z.Beek W.Wang H.Zhang Y.Zhou R.",,"Springer Science and Business Media Deutschland GmbH","21st International Conference on Web Information Systems Engineering, WISE 2020","20 October 2020 through 24 October 2020",,250469,03029743,9783030620073,,,"English","Lect. Notes Comput. Sci.",Conference Paper,"Final","",Scopus,2-s2.0-85096577205
"Kim J.-H., Ryu S., Jeong J., So D., Ban H.-J., Hong S.","57211095441;57217271715;57197116898;57190761179;57193959504;55817600100;","Impact of Satellite Sounding Data on Virtual Visible Imagery Generation Using Conditional Generative Adversarial Network",2020,"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing","13",,"9154540","4532","4541",,3,"10.1109/JSTARS.2020.3013598","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85090798696&doi=10.1109%2fJSTARS.2020.3013598&partnerID=40&md5=bea9a3e9113e0d6aa18712e58991c4ab","Department of Environment, Energy and Geoinfomatics, Sejong University, Seoul, South Korea; National Institute of Environmental Research, Incheon, South Korea; Department of Research and Development, DeepThoTh Co., Ltd., Seoul, 05006, South Korea","Kim, J.-H., Department of Environment, Energy and Geoinfomatics, Sejong University, Seoul, South Korea; Ryu, S., Department of Environment, Energy and Geoinfomatics, Sejong University, Seoul, South Korea; Jeong, J., National Institute of Environmental Research, Incheon, South Korea; So, D., Department of Environment, Energy and Geoinfomatics, Sejong University, Seoul, South Korea; Ban, H.-J., Department of Environment, Energy and Geoinfomatics, Sejong University, Seoul, South Korea; Hong, S., Department of Environment, Energy and Geoinfomatics, Sejong University, Seoul, South Korea, Department of Research and Development, DeepThoTh Co., Ltd., Seoul, 05006, South Korea","The visible band of satellite sensors is of limited use during the night due to a lack of solar reflection. This study presents an improved conditional generative adversarial networks (CGANs) model to generate virtual nighttime visible imagery using infrared (IR) multiband satellite observations and the brightness temperature difference between the two IR bands in the communication, ocean, and meteorological satellite. For the summer daytime case study with visible band imagery, our multiband CGAN model showed better statistical results [correlation coefficient (CC) = 0.952, bias = -1.752 (in a digital number (DN) unit from 0 to 255, converted from reflectance from 0 to 1), and root-mean-square-error (RMSE) = 26.851 DN] than the single-band CGAN model using a pair of visible and IR bands (CC = 0.916, bias = -4.073 DN, and RMSE = 35.349 DN). The proposed multiband CGAN model performed better than the single-band CGAN model, particularly, in convective clouds and typhoons, because of the sounding effects from the water vapor band. In addition, our multiband CGAN model provided detailed patterns for clouds and typhoons at twilight. Therefore, our results could be used for visible-based nighttime weather analysis of convective clouds and typhoons, using data from next-generation geostationary meteorological satellites. © 2008-2012 IEEE.","Clouds; conditional generative adversarial network (CGAN); deep learning; multiband; nighttime; typhoon; visible (VIS)","Clouds; Geostationary satellites; Hurricanes; Image enhancement; Mean square error; Weather satellites; Adversarial networks; Brightness temperature difference; Convective clouds; Correlation coefficient; Geostationary meteorological satellites; Root mean square errors; Satellite observations; Satellite soundings; Satellite imagery; atmospheric modeling; brightness temperature; convective cloud; machine learning; numerical model; satellite data; satellite imagery; typhoon; water vapor",,,,,"KMI2020-00510

NIER-2020-01-01-004","Manuscript received February 11, 2020; revised June 5, 2020 and July 8, 2020; accepted July 22, 2020. Date of publication August 3, 2020; date of current version August 21, 2020. This work was supported in part by the Korea Meteorological Administration Research and Development Program under Grant KMI2020-00510 and in part by a grant from the National Institute of Environment Research (NIER), funded by the Ministry of Environment (MOE) of the Republic of Korea (NIER-2020-01-01-004). (Corresponding author: Sungwook Hong.) Ji-Hye Kim, Sumin Ryu, Damwon So, and Hyun-Ju Ban are with the Department of Environment, Energy and Geoinfomatics, Sejong University, Seoul 100-011, South Korea (e-mail: jai.kim410@sejong.ac.kr; ryusm26@sju.ac.kr; dws328@sejong.ac.kr; hjban@sju.ac.kr).",,"Kim, S.-H., Suh, M.-S., Han, J.-H., Development of fog detection algorithm during nighttime using himawari-8/AHI satellite and ground observation data (2019) Asia-Pacific J. Atmospheric Sci., 55 (3), pp. 337-350. , Nov; Lee, J.-R., Chung, C.-Y., Ou, M.-L., Fog detection using geostationary satellite data: Temporally continuous algorithm (2011) Asia-Pacific J. Atmospheric Sci., 47 (2), pp. 113-122. , Mar; Schroeder, W., Oliva, P., Giglio, L., Quayle, B., Lorenz, E., Morelli, F., Active fire detection using Landsat-8/OLI data (2016) Remote. Sens. Environ., 185, pp. 210-220. , Nov; Arel, I., Rose, D.C., Karnowski, T.P., Deep machine learning-A new frontier in artificial intelligence research (2010) Ieee Comput. Intell. Mag., 5 (4), pp. 13-18. , Nov; Bengio, Y., Learning deep architectures for AI (2009) Found. Trends Mach. Learn., 2 (1), pp. 1-127. , Nov; Chi, J., Kim, H.C., Lee, S., Crawford, M.M., Deep learning based retrieval algorithm for Arctic sea ice concentration from AMSR2 passive microwave and MODIS optical data (2019) Remote. Sens. Environ., 231. , Sep; Hernández, E., Sanchez-Anguix, V., Julian, V., Palanca, J., Duque, N., Rainfall prediction:Adeep learning approach (2016) Proc. Int. Conf.Hybrid Artif. Intell. Syst., 1, pp. 151-162. , Apr; Tan, J., Deep learning convolutional neural network for the retrieval of land surface temperature from AMSR2 data in China (2019) Sensors, 19 (13). , Jul; Acharya, U.R., Bhat, P.S., Iyengar, S.S., Rao, A., Dua, S., Classification of heart rate data using artificial neural network and fuzzy equivalence relation (2003) Pattern Recognit., 36 (1), pp. 61-68. , Jan; Razavian, A.A.S., Azizpour, H., Sullivan, J., Carlsson, S., (2014) Cnn Features Off-the-shelf: An Astounding Baseline for Recognition; Liang, M., Hu, X., Recurrent convolutional neural network for object recognition (2015) Proc. Ieee Conf. Comput. Vis. Pattern Recognit., pp. 3367-3375. , Jun; Goodfellowet Al., I., (2014) Generative Adversarial Nets; Denton, E., Chintala, S., Szlam, A., Fergus, R., (2015) Deep Generative Image Models Using a Laplacian Pyramid of Adversarial Networks; Santos, C.N.D., Wadhawan, K., Zhou, B., (2017) Learning Loss Functions for Semi-supervised Learning Via Discriminative Adversarial Networks; Radford, A., Metz, L., Chintala, S., (2016) Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks; Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Smolley, S.P., (2017) Least Squares Generative Adversarial Networks; Odena, A., Olah, C., Shlens, J., (2017) Conditional Image Synthesis with Auxiliary Classifier GANs; Zhu, J.-Y., Park, T., Isola, P., Efros, A.A., (2018) Unpaired Image-toimage Translation Using Cycle-consistent Adversarial Networks; Kim, T., Cha, M., Kim, H., Lee, J.K., Kim, J., (2017) Learning to Discover Cross-domain Relations with Generative Adversarial Networks; Mirza, M., Osindero, S., (2014) Conditional Generative Adversarial Nets; Inoue, T., A cloud type classification with NOAA 7 split-window measurements (1987) J. Geophys. Res. Atmospheric, 92 (D4), pp. 3991-4000. , Apr; Lutz, H.-J., Inoue, T., Schmetz, J., Comparison of a split-window and a multi-spectral cloud classification for MODIS observations (2003) J. Meteor. Soc. Jap., 81 (3), pp. 623-631. , Jun; Hopkins, E., (2020) Weather Satellite Image Interpretation, , http://www.aos.wisc.edu/?hopkins/wx-doc/wxsatimg.htm, Accessed: May 20; Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A., (2016) Image-to-image Translation with Conditional Adversarial Networks; Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A., Context encoders: Feature learning by inpainting (2016) Proc. Conf. Comput. Vis. Pattern Recognit., pp. 2536-2544. , Nov; Zhang, R., Isola, P., Efros, A.A., (2016) Colorful Image Colorization; Wang, X., Gupta, A., (2016) Generative Image Modeling Using Style and Structure Adversarial Networks; Kiran, B.R., Thomas, D.M., Parakkal, R., (2018) An Overviewof Deep Learning Based Methods for Unsupervised and Semi-supervised Anomaly Detection in Videos; Lin, Y.-C., (2019) pix2pix-tensorflow, , https://github.com/yenchenlin/pix2pix-tensorflow, Accessed: Mar. 20; Ronneberger, O., Fischer, P., Brox, T., U-Net: Convolutional networks for biomedical image segmentation (2015) Proc. Int. Conf. Med. Image Comput. Comput.-Assist. Intervent., pp. 234-241. , May; Li, C., Wand, M., Precomputed real-time texture synthesis with markovian generative adversarial networks (2016) Proc. Eur. Conf. Comput. Vis., pp. 702-716. , Apr; Michelsanti, D., Tan, Z.-H., (2017) Conditional Generative Adversarial Networks for Speech Enhancement and Noise-robust Speaker Verification; Kim, K., Nighttime reflectance generation in the visible band of satellites (2019) Remote Sens., 11 (18). , Sep; Kim, Y., Hong, S., Deep learning-generated nighttime reflectance and daytime radiance of themidwave infrared band of a geostationary satellite (2019) Remote Sens., 11 (22). , Nov; Kim, M.-S., Hong, S.-G., Seasonal change of the frequency percentage of cloud occurrence according to its type, height and amount in Korea (1991) Asia-Pac. J. Atmospheric Sci., 27 (4), pp. 353-364. , Dec","Hong, S.; Department of Environment, South Korea; email: sesttiya@deep-thoth.org",,,"Institute of Electrical and Electronics Engineers",,,,,19391404,,,,"English","IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.",Article,"Final","All Open Access, Gold",Scopus,2-s2.0-85090798696
"Wu Z., Wu X., Zhu Y.","57194799907;56129462300;56025861600;","Structured illumination-based phase retrieval via Generative Adversarial Network",2020,"Progress in Biomedical Optics and Imaging - Proceedings of SPIE","11249",,"112490L","","",,3,"10.1117/12.2547551","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85082687680&doi=10.1117%2f12.2547551&partnerID=40&md5=3624842b0614fc96f998c43bf69a49d2","Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA  24060, United States","Wu, Z., Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA  24060, United States; Wu, X., Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA  24060, United States; Zhu, Y., Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA  24060, United States","Structured-illumination (SI) is used for quantitative phase retrieval for improved contrast and sensitivity. However, the nonlinear nature of SI-based phase retrieval process, such as the spatial frequency biases and mixture of different spatial frequency components, usually leads to phase aberrations, in particular in the high spatial frequency components. Recent studies show that nonlinear inversion problems can be efficiently represented by deep neural networks in an end-to-end framework. In this study, we present a deep learning framework for SI-based quantitative phase imaging via the Conditional Generative Adversarial Network (cGANs). A series of structured images paired with the corresponding ground truth of phase images are used to train two competing networks of generator and discriminator. We demonstrate that the GAN-based approach produces sharp and accurate phase image and the structured illumination pattern simultaneously based on our simulation. © 2020 SPIE.","Computational imaging; Deep neural networks; Generative Adversarial Network; Phase retrieval; Structured illumination","Deep learning; Adversarial networks; Computational imaging; High spatial frequency; Learning frameworks; Non linear inversion; Phase retrieval; Quantitative phase imaging; Structured illumination; Deep neural networks",,,,,,,,"Wicker, K., Mandula, O., Best, G., Fiolka, R., Heintzmann, R., Phase optimisation for structured illumination microscopy (2013) Opt. Express, 21, p. 2032; Pospisil, J., Fliegel, K., Klima, M., Phase estimation of illumination pattern in structured illumination microscopy (2017) 2017 27th Int. Conf. Radioelektronika, RADIOELEKTRONIKA 2017, pp. 1-4; Lal, A., Shan, C., Xi, P., Structured illumination microscopy image reconstruction algorithm (2016) IEEE J. Sel. Top. Quantum Electron., 22, pp. 50-63; Jaganathan, K., Eldar, Y.C., Hassibi, B., (2015) Phase Retrieval: An Overview of Recent Developments, pp. 1-24; Liu, Y.J., (2008) Phase Retrieval in X-ray Imaging Based on Using Structured Illumination, pp. 2-6; Zhu, Y., Zhang, Z., Barbastathis, G., Phase imaging for absorptive phase objects using hybrid uniform and structured illumination Transport of Intensity Equation (2014) Opt. Express, 22, p. 28966; Chakrova, N., Rieger, B., Stallinga, S., Deconvolution methods for structured illumination microscopy (2016) J. Opt. Soc. Am. A, 33, p. B12; Wu, Z., Zhu, Y., (2017) Title : Superresolution Phase Retrieval from Non-sinusoidal Structure Illumination Authors : Yunhui Zhu and Ziling Wu Event Name : Imaging Systems and Applications Year : Superresolution Phase Retrieval from Non-sinusoidal Structure Illumination; Wang, Z., Ding, H., Popescu, G., Spatial light interference microscopy (SLIM) (2010) Opt. InfoBase Conf. Pap., 19, pp. 2643-2648; Gerchberg, R.W., Saxton, W., A practical algorithm for the determination of phase from image and diffraction plane pictures (1972) Optik (Stuttg)., 35, pp. 237-246; Fienup, J.R., Reconstruction of an object from the modulus of its Fourier transform (1978) Opt. Lett., 3, p. 27; Yang, G., Dong, B., Gu, B., Zhuang, J., Ersoy, O.K., Gerchberg-saxton and yang-gu algorithms (1994) Appl. Opt., 33, pp. 209-218; Gustafsson, M.G.L., Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution (2005) Proc. Natl. Acad. Sci. U. S. A., 102, pp. 13081-13086; Zhu, Y., Li, X., Wu, Z., Li, X., Zhu, Y., (2017) Illumination for Microvessel Characterization Authors, , Event Name : Frontiers in Optics Year : Title : Quantitative X-ray ultra-small angle scattering retrieval with structured illumination for microvessel characterization; Kellman, M.R., Bostan, E., Repina, N., Waller, L., (2015) Physics-based Learned Design : Optimized Coded-Illumination for Quantitative Phase Imaging, 14, pp. 1-10; Kappeler, A., Ghosh, S., Holloway, J., Cossairt, O., Katsaggelos, A., PTYCHNET : CNN BASED FOURIER PTYCHOGRAPHY; Goy, A., High-resolution limited-angle phase tomography of dense layered objects using deep neural networks (2019) Proc. Natl. Acad. Sci. U. S. A., 116, pp. 19848-19856; Ren, Z., Xu, Z., Lam, E.Y., (2019) End-to-end Deep Learning Framework for Digital Holographic Reconstruction, 1, pp. 1-12; Sinha, A., Lee, J., Li, S., Barbastathis, G., (2017) Lensless Computational Imaging Through Deep Learning, 4; Rivenson, Y., Zhang, Y., Gunaydin, H., Teng, D., Ozcan, A., (2017) Phase Recovery and Holographic Image Reconstruction Using Deep Learning in Neural Networks, pp. 1-13; Jo, Y., Quantitative Phase Imaging and Artificial Intelligence : A Review (2019) IEEE J. Sel. Top. Quantum Electron., 25, pp. 1-14; Barbastathis, G., Ozcan, A., Situ, G., On the use of deep learning for computational imaging (2019) Optica, 6, p. 921; Deng, M., Li, S., Barbastathis, G., Learning to Synthesize: Splitting and Recombining Low and High Spatial Frequencies for Image Recovery; Ying, X., X2ct-gan: Reconstructing ct from biplanar x-rays with generative (2019) Adversarial Networks, pp. 10619-10628; Rubin, J., Abulnaga, S.M., (2019) CT-To-MR Conditional Generative Adversarial Networks for Ischemic Stroke Lesion Segmentation, pp. 1-11; Paganin, D., Nugent, K.A., Noninterferometric phase imaging with partially coherent light (1998) Phys. Rev. Lett., 80, pp. 2586-2589; Dennis, M.R., O'Holleran, K., Padgett, M.J., Chapter 5 singular optics: Optical vortices and polarization singularities (2009) Progress in Optics, 53. , Elsevier B.V; Nasrollahi, K., Moeslund, T.B., Super-resolution: A comprehensive survey (2014) Machine Vision and Applications, 25; Rivenson, Y., Zhang, Y., Günaydin, H., Teng, D., Ozcan, A., Phase recovery and holographic image reconstruction using deep learning in neural networks (2018) Light Sci. Appl., 7, p. 17141; Knyaz, V.A., Kniaz, V.V., Remondino, F., Image-to-voxel model translation with conditional adversarial networks (2019) Lect. Notes Comput. Sci. (Including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics) 11129 LNCS, pp. 601-618; Kingma, D.P., Ba, J.L., Adam: A method for stochastic optimization (2015) 3rd Int. Conf. Learn. Represent. ICLR 2015-Conf. Track Proc, pp. 1-15","Zhu, Y.; Department of Electrical and Computer Engineering, United States; email: yunhuiz@vt.edu","Liu Y.Popescu G.Park Y.","Phi Optics, Inc.;The Society of Photo-Optical Instrumentation Engineers (SPIE);Tomocube, Inc.","SPIE","Quantitative Phase Imaging VI 2020","1 February 2020 through 4 February 2020",,158340,16057422,9781510632615,,,"English","Progr. Biomed. Opt. Imaging Proc. SPIE",Conference Paper,"Final","",Scopus,2-s2.0-85082687680
"Ngxande M., Tapamo J.-R., Burke M.","57202813365;55916165800;36099859700;","Bias remediation in driver drowsiness detection systems using generative adversarial networks",2020,"IEEE Access","8",,"9042231","55592","55601",,6,"10.1109/ACCESS.2020.2981912","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85082678877&doi=10.1109%2fACCESS.2020.2981912&partnerID=40&md5=15dc6908ac7a41b5a5b9ed13878da784","School of Engineering, University of Kwa-Zulu Natal, Durban, 4041, South Africa; School of Informatics, Institute of Perception, Action and Behaviour, University of Edinburgh, Edinburgh, EH8 9AB, United Kingdom","Ngxande, M., School of Engineering, University of Kwa-Zulu Natal, Durban, 4041, South Africa; Tapamo, J.-R., School of Engineering, University of Kwa-Zulu Natal, Durban, 4041, South Africa; Burke, M., School of Informatics, Institute of Perception, Action and Behaviour, University of Edinburgh, Edinburgh, EH8 9AB, United Kingdom","Datasets are crucial when training a deep neural network. When datasets are unrepresentative, trained models are prone to bias because they are unable to generalise to real world settings. This is particularly problematic for models trained in specific cultural contexts, which may not represent a wide range of races, and thus fail to generalise. This is a particular challenge for driver drowsiness detection, where many publicly available datasets are unrepresentative as they cover only certain ethnicity groups. Traditional augmentation methods are unable to improve a model's performance when tested on other groups with different facial attributes, and it is often challenging to build new, more representative datasets. In this paper, we introduce a novel framework that boosts the performance of detection of drowsiness for different ethnicity groups. Our framework improves Convolutional Neural Network (CNN) trained for prediction by using Generative Adversarial networks (GAN) for targeted data augmentation based on a population bias visualisation strategy that groups faces with similar facial attributes and highlights where the model is failing. A sampling method selects faces where the model is not performing well, which are used to fine-tune the CNN. Experiments show the efficacy of our approach in improving driver drowsiness detection for under represented ethnicity groups. Here, models trained on publicly available datasets are compared with a model trained using the proposed data augmentation strategy. Although developed in the context of driver drowsiness detection, the proposed framework is not limited to the driver drowsiness detection task, but can be applied to other applications. © 2013 IEEE.","CNN; GAN; Population bias; visualisation","Deep neural networks; Population statistics; Visualization; Adversarial networks; Augmentation methods; Cultural context; Data augmentation; Driver drowsiness; Population bias; Real world setting; Under-represented; Convolutional neural networks",,,,,,"This work was supported by the University of Kwa-Zulu Natal.",,"https://www.who.int/violence_injury_prevention/road_safety_status/2018/English-Summary-GSRRS2018.pdf, Global Status Report on Road Safety 2018. Accessed: Apr. 19, 2019; (2019), http://driverfatigue.co.uk/, Accessed: Feb. 25; https://www.who.int/violence_injury_prevention/road_safety_status/2018/en/, Global Status Report on Road Safety 2018. Accessed: Mar. 3, 2019; https://www.wheels24.co.za, Africa Has 2% of World'S Cars But 20% of Road Deaths'-First Safety Observatory to Curb Horrendous Death Toll. Accessed: Mar. 5, 2019; Wierwille, W., Knipling, R., Vehicle-based drowsy driver detection: Current status and future prospects (1994) Proc. IVHS Amer., pp. 245-256. , Apr; Sahayadhas, A., Sundaraj, K., Murugappan, M., Detecting driver drowsiness based on sensors: A review (2012) Sensors, 12 (12), pp. 16937-16953; Luthra, A., (2016) Echo Made Easy, , Ahmedabad, Gujarat: JP Medical; Drewes, C., Electromyography: Recording electrical signals from human muscle (2000) Tested Stud. Lab. Teach. Assoc. Biol. Lab. Educ. (ABLE), 21, pp. 248-270; Folane, N.R., Autee, R.M., EEG based brain controlled wheelchair for physically challenged people (2016) Int. J. Innov. Res. Comput. Commun. Eng., 4 (6), pp. 2257-2263; Awais, M., Badruddin, N., Drieberg, M., A hybrid approach to detect driver drowsiness utilizing physiological signals to improve system performance and wearability (2017) Sensors, 17 (9), p. 1991; Li, M., Meng, H.-L., A method of driver fatigue detection based on multi-features (2015) Int. J. Signal Process., Image Process. Pattern Recognit., 8 (10), pp. 107-114. , Oct; Rajput, M.V., Bakal, J.W., Execution scheme for driver drowsiness detection using yawning feature (2013) Int. J. Comput. Appl., 62 (6), pp. 6-11; De Naurois, C.J., Bourdin, C., Stratulat, A., Diaz, E., Vercher, J.-L., Detection and prediction of driver drowsiness using artificial neural network models (2019) Accident Anal. Prevention, 126, pp. 95-104. , May; Ngxande, M., Tapamo, J.-R., Burke, M., Driver drowsiness detection using behavioral measures and machine learning techniques: A review of state-of-art techniques (2017) Pattern Recognit. Assoc. South Afr. Robot. Mechatronics (PRASA-RobMech), , Bloemfontein, South Africa, Nov; Shima, R., Yunan, H., Fukuda, O., Okumura, H., Arai, K., Bu, N., Object classification with deep convolutional neural network using spatial information (2017) Proc. Int. Conf. Intell. Inform. Biomed. Sci. (ICI-IBMS), pp. 135-139. , Nov; Jangid, S., Bhatnagar, P.S., Semantic image segmentation using deep convolutional neural networks and super-pixels (2018) Int. J. Appl. Eng. Res., 13 (20), pp. 14657-14663; Li, D., Chen, W., Object tracking with convolutional neural networks and kernelized correlation filters (2017) Proc. 29th Chin. Control Decis. Conf. (CCDC), pp. 1039-1044. , May; Ngxande, M., Tapamo, J.-R., Burke, M., (2019) Detecting Inter-sectional Accuracy Differences in Driver Drowsiness Detection Algorithms, , http://arxiv.org/abs/1904.12631; Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Proc. Adv. Neural Inf. Process. Syst, pp. 2672-2680; Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A., (2016) Image-to-image Translation with Conditional Adversarial Networks, , http://arxiv.org/abs/1611.07004; Ngiam, J., Peng, D., Vasudevan, V., Kornblith, S., Le, Q.V., Pang, R., (2018) Domain Adaptive Transfer Learning with Specialist Models, , http://arxiv.org/abs/1811.07056; Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Loy, C.C., Tang, X., (2018) ESRGAN: Enhanced Super-resolution Generative Adversarial Networks, , http://arxiv.org/abs/1809.00219; Bodnar, C., (2018) Text to Image Synthesis Using Generative Adversarial Networks, , http://arxiv.org/abs/1805.00676; Arjovsky, M., Chintala, S., Bottou, L., Wasserstein GAN (2017) Proc. 34th ICML, 70, pp. 214-223. , Sydney, NSW, Australia, Aug; Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A., (2017) Improved Training of Wasserstein GANs, , http://arxiv.org/abs/1704.00028; Radford, A., Metz, L., Chintala, S., (2015) Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, , http://arxiv.org/abs/1511.06434; Mirza, M., Osindero, S., (2014) Conditional Generative Adversarial Nets, , http://arxiv.org/abs/1411.1784; Gupta, R., (2019) Data Augmentation for Low Resource Sentiment Analysis Using Generative Adversarial Networks, , http://arxiv.org/abs/1902.06818; Mok, T.C.W., Chung, A.C.S., (2018) Learning Data Augmentation for Brain Tumor Segmentation with Coarse-to-Fine Generative Adversarial Networks, , http://arxiv.org/abs/1805.11291; Wu, E., Wu, K., Cox, D., Lotter, W., (2018) Conditional Infilling GANs for Data Augmentation in Mammogram Classification, , http://arxiv.org/abs/1807.08093; Antoniou, A., Storkey, A., Edwards, H., Augmenting image classifiers using data augmentation generative adversarial networks (2018) Proc. Int. Conf. Artif. Neural Netw. Cham, pp. 594-603. , Switzerland: Springer, Oct; Stewart, G.W., On the early history of the singular value decomposition (1993) SIAM Rev., 35 (4), pp. 551-566. , Dec; Parris, J., Wilber, M., Hein, B., Rara, H., El-Barkouky, A., Farag, A., Movellan, J., Boult, T.E., Face and eye detection on hard datasets (2011) Proc. Int. Joint Conf. Biometrics (IJCB), pp. 1-10. , Oct; Shadowen, N., Ethics and bias in machine learning: A technical study of what makes us good (2019) ' in the Transhumanism Handbook. Cham, pp. 247-261. , Switzerland: Springer; Rhue, L., (2019) Emotion-Reading Tech Fails the Racial Bias Test, , The Conversation; Raji, I.D., Gebru, T., Mitchell, M., Buolamwini, J., Lee, J., Denton, E., (2020) Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditing, , https://arxiv.org/abs/2001.00964; Buolamwini, J., Gebru, T., Gender shades: Intersectional accuracy disparities in commercial gender classification (2018) Proc. 1st Conf. Fairness, Accountability Transparency, pp. 1-10. , New York, NY, USA, Feb; Garvie, C., Bedoya, A., Frankle, J., (2016) The Perpetual Line-Up. Unregulated Police Face Recognition in America, , Washington, DC, USA: Georgetown Law Center Privacy &Technology, Oct; De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Geyik, S., Chouldechova, A., Tauman Kalai, A., (2019) Bias in Bios: A Case Study of Semantic Representation Bias in A High-stakes Setting, , http://arxiv.org/abs/1901.09451; Benthall, S., Haynes, B.D., Racial categories in machine learning (2019) Proc. Conf. Fairness, Accountability, Transparency, pp. 289-298. , Atlanta, GA, USA; Abiteboul, S., Issues in ethical data management (2017) 19th Int. Symp. Princ. Pract. Declarative Program., , New York, NY, USA; Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., Choo, J., (2017) Star-GAN: Unified Generative Adversarial Networks for Multi-domain Imageto-Image Translation, , http://arxiv.org/abs/1711.09020; Ngxande, M., Tapamo, J., Burke, M., DepthwiseGANs: Fast training generative adversarial networks for realistic image synthesis (2019) Proc. Southern Afr. Universities Power Eng. Conf./Robot. Mechatronics/Pattern Recognit. Assoc. South Afr. SAUPEC/RobMech/PRASA, pp. 111-116. , Bloemfontein, South Africa, Jan; Ulyanov, D., Vedaldi, A., Lempitsky, V., (2016) Instance Normalization: The Missing Ingredient for Fast Stylization, , http://arxiv.org/abs/1607.08022; Li, C., Wand, M., (2016) Precomputed Real-time Texture Synthesis with Markovian Generative Adversarial Networks, , http://arxiv.org/abs/1604.04382; He, K., Zhang, X., Ren, S., Sun, J., (2015) Deep Residual Learning for Image Recognition, , http://arxiv.org/abs/1512.03385; Bashivan, P., Rish, I., Yeasin, M., Codella, N., (2015) Learning Representations from EEG with Deep Recurrent-convolutional Neural Networks, , http://arxiv.org/abs/1511.06448; Kingma, D.P., Ba, J., (2014) Adam: A Method for Stochastic Optimization, , http://arxiv.org/abs/1412.6980; http://cv.cs.nthu.edu.tw/php/callforpaper/datasets/DDD/, NTHU CVlab-Driver Drowsiness Detection Dataset. Accessed: Aug. 3, 2018; Massoz, Q., Langohr, T., François, C., Verly, J.G., The ULg multimodality drowsiness database (called DROZY) and examples of use (2016) Proc. IEEE Winter Conf. Appl. Comput. Vis. (WACV), pp. 1-7. , Mar; http://parnec.nuaa.edu.cn/xtan/data/ClosedEyeDatabases.html, The Closed Eyes in the Wild (CEW) Dataset. Accessed: Apr. 19, 2018","Ngxande, M.; School of Engineering, South Africa; email: mngxande@gmail.com",,,"Institute of Electrical and Electronics Engineers Inc.",,,,,21693536,,,,"English","IEEE Access",Article,"Final","All Open Access, Gold, Green",Scopus,2-s2.0-85082678877
"Ryu S., Koo S., Yu H., Lee G.G.","9041008800;56516215900;24400252600;7404853194;","Out-of-domain detection based on generative adversarial network",2020,"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018",,,,"714","718",,7,,"https://www.scopus.com/inward/record.uri?eid=2-s2.0-85081721805&partnerID=40&md5=74bce38025c3cb38dcba60c989380ac0","Language Understanding Lab, Artificial Intelligence Center, Samsung Research; Computer Science and Engineering Department, Pohang University of Science and Technology, South Korea","Ryu, S., Language Understanding Lab, Artificial Intelligence Center, Samsung Research, Computer Science and Engineering Department, Pohang University of Science and Technology, South Korea; Koo, S., Computer Science and Engineering Department, Pohang University of Science and Technology, South Korea; Yu, H., Computer Science and Engineering Department, Pohang University of Science and Technology, South Korea; Lee, G.G., Computer Science and Engineering Department, Pohang University of Science and Technology, South Korea","The main goal of this paper is to develop out-of-domain (OOD) detection for dialog systems. We propose to use only in-domain (IND) sentences to build a generative adversarial network (GAN) of which the discriminator generates low scores for OOD sentences. To improve basic GANs, we apply feature matching loss in the discriminator, use domain-category analysis as an additional task in the discriminator, and remove the biases in the generator. Thereby, we reduce the huge effort of collecting OOD sentences for training OOD detection. For evaluation, we experimented OOD detection on a multi-domain dialog system. The experimental results showed the proposed method was most accurate compared to the existing methods. © 2018 Association for Computational Linguistics",,"Adversarial networks; Dialog systems; Domain detections; Feature matching; Multi domains; Natural language processing systems",,,,,"Ministry of Science, ICT and Future Planning, MSIP: IITP-2018-2015-0-00742

Institute for Information and Communications Technology Promotion, IITP

Ministry of Science and ICT, South Korea, MSIT","This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the Grand Information Technology Research Center support program (IITP-2018-2015-0-00742) supervised by the IITP(Institute for Information & communications Technology Promotion)",,"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Devin, M., Vanhoucke, W., (2015) TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems, , Software tensor-flow.org; Breunig, M.M., Kriegel, H.-P., Ng, R.T., Sander, J., LOF: Identifying density-based local outliers (2000) Proceedings of ACM SIGMOD; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Proceedings of NIPS; Hakkani-Tur, D., Tur, G., Celikyilmaz, A., Chen, Y.-N., Gao, J., Deng, L., Wang, Y.-Y., Multi-domain joint semantic frame parsing using bi-directional RNN-LSTM (2016) Proceedings of Interspeech; Jiang, R., Banchs, R.E., Kim, S., Yeo, K.H., Niswar, A., Li, H., Web-based multimodal multi-domain spoken dialogue system (2014) Proceedings of IWSDS; Kingma, D., Ba, J., ADaM: A method for stochastic optimization (2015) Proceedings of ICLR; Lane, I., Kawahara, T., Matsui, T., Nakamura, S., Out-of-domain utterance detection using classification confidences of multiple topics (2007) IEEE/ACM Trans. Audio, Speech, Language Process., 15, pp. 150-161; Lee, D., Jeong, M., Kim, K., Ryu, S., Lee, G.G., Unsupervised spoken language understanding for a multi-domain dialog system (2013) IEEE/ACM Trans. Audio, Speech, Language Process., 21, pp. 2451-2464; Ryu, S., Kim, S., Choi, J., Yu, H., Lee, G.G., Neural sentence embedding using only in-domain sentences for out-of-domain sentence detection in dialog systems (2017) Pattern Recogn. Lett., 88, pp. 26-32; Ryu, S., Song, J., Koo, S., Kwon, S., Lee, G.G., Detecting multiple domains from users utterance in spoken dialog system (2015) Proceedings of IWSDS; Salimans, T., Goodfellow, I.J., Zaremba, W., Cheung, V., Radford, A., Chen, X., Improved techniques for training GANs (2016) Proceedings of NIPS; Schölkopf, B., Platt, J.C., Shawe-Taylor, J.C., Smola, A.J., Williamson, R.C., Estimating the support of a high-dimensional distribution (2001) Neural Comput, 13, pp. 1443-1471; Seon, C.-N., Lee, H., Kim, H., Seo, J., Improving domain action classification in goal-oriented dialogues using a mutual retraining method (2014) Pattern Recogn. Lett., 45",,"Riloff E.Chiang D.Hockenmaier J.Tsujii J.","Apple;Bloomberg;et al.;Facebook;Google;salesforce","Association for Computational Linguistics","2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018","31 October 2018 through 4 November 2018",,158085,,9781948087841,,,"English","Proc. Conf. Empir. Methods Nat. Lang. Process., EMNLP",Conference Paper,"Final","",Scopus,2-s2.0-85081721805
"Wu B., Liu L., Yang Y., Zheng K., Wang X.","56449782000;57212143508;54379256800;23391663200;36602029300;","Using improved conditional generative adversarial networks to detect social bots on Twitter",2020,"IEEE Access","8",,"9006873","36664","36680",,8,"10.1109/ACCESS.2020.2975630","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85081131120&doi=10.1109%2fACCESS.2020.2975630&partnerID=40&md5=fb9430a95338e7d7b363b7cbaccae936","School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, 100876, China; Faculty of Information Technology, Beijing University of Technology, Beijing, 100124, China","Wu, B., School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, 100876, China; Liu, L., School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, 100876, China; Yang, Y., School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, 100876, China; Zheng, K., School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, 100876, China; Wang, X., Faculty of Information Technology, Beijing University of Technology, Beijing, 100124, China","The detection and removal of malicious social bots in social networks has become an area of interest in industry and academia. The widely used bot detection method based on machine learning leads to an imbalance in the number of samples in different categories. Classifier bias leads to a low detection rate of minority samples. Therefore, we propose an improved conditional generative adversarial network (improved CGAN) to extend imbalanced data sets before applying training classifiers to improve the detection accuracy of social bots. To generate an auxiliary condition, we propose a modified clustering algorithm, namely, the Gaussian kernel density peak clustering algorithm (GKDPCA), which avoids the generation of data-augmentation noise and eliminates imbalances between and within social bot class distributions. Furthermore, we improve the CGAN convergence judgment condition by introducing the Wasserstein distance with a gradient penalty, which addresses the model collapse and gradient disappearance in the traditional CGAN. Three common oversampling algorithms are compared in experiments. The effects of the imbalance degree and the expansion ratio of the original data on oversampling are studied, and the improved CGAN performs better than the others. Experimental results comparing with three common oversampling algorithms show that the improved CGAN achieves the higher evaluation scores in terms of F1-score, G-mean and AUC. © 2013 IEEE.","conditional generative adversarial networks; data augmentation; imbalanced data; Social bot detection; supervised classification","Botnet; Classification (of information); Supervised learning; Adversarial networks; Data augmentation; Imbalanced data; Social bots; Supervised classification; Clustering algorithms",,,,,"Natural Science Foundation of Beijing Municipality: 4202002

National Basic Research Program of China (973 Program): 2017YFB0802703","This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFB0802703, and in part by the Beijing Natural Science Foundation under Grant 4202002.",,"Clement, J., (2019) Facebook: Number of Monthly Active Users Worldwide 2008-2019, , https://www.statista.com/statistics/264810/number-of-monthly-active-facebook-users-worldwide/, Nov; Clement, J., (2019) Twitter: Number of Monthly Active Users 2010-2019, , https://www.statista.com/statistics/282087/number-of-monthly-active-twitter-users/, Aug; Aslam, S., (2019) Twitter by the Numbers: Stats, Demographics & Fun Facts, , https://www.omnicoreagency.com/twitterstatistics/, Sep; Ferrara, E., Varol, O., Davis, C., Menczer, F., Flammini, A., The rise of social bots (2016) Commun. ACM, 59 (7), pp. 96-104. , Jun; Cassa, C., Chunara, R., Mandl, K., Brownstein, J.S., Twitter as a sentinel in emergency situations: Lessons from the boston marathon explosions (2013) PLoS Currents; Conover, M., Ratkiewicz, J., Francisco, M., Gonçalves, B., Menczer, F., Flammini, A., Political polarization on Twitter (2011) Proc. 5th Int. AAAI Conf. Weblogs Social Media, pp. 89-96. , Jul; (2018), https://www.washingtonpost.com/technology/2018/07/06/twitter-is-sweeping-out-fake-accounts-likenever-before-putting-user-growth-risk/?utm_term=.5d2f229e7794, Accessed: Jul. 6; Abokhodair, N., Yoo, D., McDonald, D.W., Dissecting a social botnet: Growth, content and Influence in Twitter (2015) Proc. 18th ACM Conf. Comput. Supported Cooperat. Work Social Comput. (CSCW), pp. 839-851; Subrahmanian, V.S., Azaria, A., Durst, S., Kagan, V., Galstyan, A., Lerman, K., Zhu, L., Menczer, F., The DARPA Twitter bot challenge (2016) Computer, 49 (6), pp. 38-46. , Jun; Lingam, G., Rout, R.R., Somayajulu, D.V.L.N., Detection of social botnet using a trust model based on spam content in Twitter network (2018) Proc. IEEE 13th Int. Conf. Ind. Inf. Syst. (ICIIS), pp. 280-285. , Dec; Lemley, J., Bazrafkan, S., Corcoran, P., Smart augmentation learning an optimal data augmentation strategy (2017) IEEE Access, 5, pp. 5858-5869. , Mar; Charalambous, C., Bharath, A., A data augmentation methodology for training machine/deep learning gait recognition algorithms (2016) Proc. Brit. Mach. Vis. Conf., pp. 1101-11012. , Sep; Alothali, E., Zaki, N., Mohamed, E.A., Alashwal, H., Detecting social bots on Twitter: A literature review (2018) Proc. Int. Conf. Innov. Inf. Technol. (IIT), pp. 175-180. , Nov; Chu, Z., Gianvecchio, S., Wang, H., Jajodia, S., Who is tweeting on Twitter: Human, bot, or cyborg? (2010) Proc. 26th Annu. Comput. Secur. Appl. Conf., pp. 21-30. , Dec; Varol, O., Ferrara, E., Clayton Davis, A., Menczer, F., Flammini, A., Online human-bot interactions: Detection, estimation, and characterization (2017) Proc. 11th Int. AAAI Conf. Web Social Media, pp. 280-289. , May; Yang, Z., Wilson, C., Wang, X., Gao, T., Ben Zhao, Y., Dai, Y., Uncovering social network sybils in the wild (2011) Proc. ACM SIGCOMM Conf. Internet Meas. Conf., pp. 259-268. , Nov; Gilani, Z., Farahbakhsh, R., Tyson, G., Wang, L., Crowcroft, J., Of bots and humans (on Twitter) (2017) Proc. IEEE/ACM Int. Conf. Adv. Social Netw. Anal. Mining (ASONAM), pp. 349-354. , Aug; Zafarani, R., Liu, H., 10 bits of surprise: Detecting malicious users with minimum information (2015) Proc. 24th ACMInt. Conf. Inf. Knowl. Manage., pp. 423-431. , Oct; Clark, E.M., Williams, J.R., Jones, C.A., Galbraith, R.A., Danforth, C.M., Dodds, P.S., Sifting robotic from organic text: A natural language approach for detecting automation on Twitter (2016) J. Comput. Sci., 16, pp. 1-7. , Sep; Van Der Walt, E., Eloff, J., Using machine learning to detect fake identities: Bots vs humans (2018) IEEE Access, 6, pp. 6540-6549; Loyola-González, O., Monroy, R., Rodríguez, J., López-Cuevas, A., Mata-Sánchez, J.I., Contrast pattern-based classification for bot detection on Twitter (2019) IEEE Access, 7, pp. 45800-45817; Wang, G., Konolige, T., Wilson, C., Wang, X., Zheng, H., Zhao, B.Y., You are how you click: Clickstream analysis for sybil detection (2013) Proc. 22nd USENIX Secur. Symp., pp. 241-256. , Aug; Shi, P., Zhang, Z., Choo, K.-K.-R., Detecting malicious social bots based on clickstream sequences (2019) IEEE Access, 7, pp. 28855-28862; Cao, Q., Sirivianos, M., Yang, X., Pregueiro, T., Aiding the detection of fake accounts in large scale social online services (2012) Proc. 10th USENIX Symp. Networked Syst. Des. Implement., pp. 197-210. , Apr; Cai, C., Li, L., Zengi, D., Behavior enhanced deep bot detection in social media (2017) Proc. IEEE Int. Conf. Intell. And Secur. Inform. (ISI), pp. 128-130. , Jul; Chavoshi, N., Hamooni, H., Mueen, A., DeBot: Twitter bot detection via warped correlation (2016) Proc. IEEE 16th Int. Conf. Data Mining (ICDM), pp. 817-822. , Dec; Kudugunta, S., Ferrara, E., Deep neural networks for bot detection (2018) Inf. Sci., 467, pp. 312-322. , Oct; Beutel, A., Xu, W., Guruswami, V., Palow, C., Faloutsos, C., Copy-Catch: Stopping group attacks by spotting lockstep behavior in social networks (2013) Proc. 22nd Int. Conf. World Wide Web (WWW), pp. 119-130; Cresci, S., Pietro, R.D., Petrocchi, M., Spognardi, A., Tesconi, M., Social fingerprinting: Detection of spambot groups through dnainspired behavioral modeling (2018) IEEE Trans. Dependable Secure Comput., 15 (4), pp. 561-576. , Jul; Sun, Y., Wong, A.K., Kamel, M.S., Classification of imbalanced data: A review (2009) Int. J. Pattern Recognit. Artif. Intell., 23 (4), pp. 687-719; Kaur, H., Pannu, H.S., Malhi, A.K., A systematic review on imbalanced data challenges in machine learning: Applications and solutions (2019) ACM Comput. Surv., 52 (4), pp. 1-36. , Aug; Xu, X., Chen, W., Sun, Y., Over-sampling algorithm for imbalanced data classification (2019) J. Syst. Eng. Electron., 30 (6), pp. 1182-1191; Haixiang, G., Yijing, L., Shang, J., Mingyun, G., Yuanyue, H., Bing, G., Learning from class-imbalanced data: Review of methods and applications (2017) Expert Syst. Appl., 73, pp. 220-239. , May; Longadge, R., Snehlata Dongre, S., Malik, L., Class imbalance problem in data mining: Review (2013) Int. J. Comput. Sci. Netw., 2, pp. 83-87. , Feb; Thabtah, F., Hammoud, S., Kamalov, F., Gonsalves, A., Data imbalance in classification: Experimental evaluation (2020) Inf. Sci., 513, pp. 429-441. , Mar; Buda, M., Maki, A., Mazurowski, M.A., A systematic study of the class imbalance problem in convolutional neural networks (2018) Neural Netw., 106, pp. 249-259. , Oct; López, V., Fernández, A., García, S., Palade, V., Herrera, F., An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics (2013) Inf. Sci., 250, pp. 113-141. , Nov; Hart, P.E., The condensed nearest neighbor rule (1968) IEEE Trans. Inf. Theory, 14 (3), pp. 515-516. , May; Tomek, I., Two modifications of CNN (1976) IEEE Trans. Syst., Man, Cybern., SMC-6 (11), pp. 769-772. , Nov; Kubat, M., Matwin, S., Addressing the curse of imbalanced training sets: One-sided selection (1997) Proc. Int. Conf. Mach. Learn., pp. 179-186. , Jul; Laurikkala, J., Improving identification of difficult small classes by balancing class distribution (2001) Artificial Intelligence in Medicine, pp. 63-66. , Berlin, Germany: Springer, Jun; Wilson, D.L., Asymptotic properties of nearest neighbor rules using edited data (1972) IEEE Trans. Syst., Man, Cybern., SMC-2 (3), pp. 408-421. , Jul; Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P., SMOTE: Synthetic minority over-sampling technique (2002) J. Artif. Intell. Res., 16, pp. 321-357. , Jun; Barua, S., Islam, M.M., Yao, X., Murase, K., MWMOTE-majority weighted minority oversampling technique for imbalanced data set learning (2014) IEEE Trans. Knowl. Data Eng., 26 (2), pp. 405-425. , Feb; Han, H., Wang, W., Mao, B., Borderline-SMOTE: A new oversampling method in imbalanced data sets learning (2005) Proc. Int. Conf. Intell. Comput., 3644, pp. 878-887; Bunkhumpornpat, C., Sinapiromsaran, K., Lursinsap, C., Safe-levelsmote: Safe-level-synthetic minority over-sampling technique for handling the class imbalanced problem (2009) Proc. Pacific-Asia Conf. Knowl. Dis-covery Data Mining, 5476, pp. 475-482; Douzas, G., Bacao, F., Last, F., Improving imbalanced learning through a heuristic oversampling method based on k-means and SMOTE (2018) Inf. Sci., 465, pp. 1-20. , Oct; Zhu, T., Lin, Y., Liu, Y., Synthetic minority oversampling technique for multiclass imbalance problems (2017) Pattern Recognit., 72, pp. 327-340. , Dec; Zhu, T., Lin, Y., Liu, Y., Zhang, W., Zhang, J., Minority oversampling for imbalanced ordinal regression (2019) Knowl.-Based Syst., 166, pp. 140-155. , Feb; Zhang, C., Chen, Y., Liu, X., Zhao, X., Abstention-SMOTE: An oversampling approach for imbalanced data classification (2017) Proc. Int. Conf. Inf. Technol., pp. 17-21. , Dec; Jiang, K., Lu, J., Xia, K., A novel algorithm for imbalance data classification based on genetic algorithm improved SMOTE (2016) Arabian J. Sci. Eng., 41 (8), pp. 3255-3266. , May; Prusty, M.R., Jayanthi, T., Velusamy, K., Weighted-SMOTE: A modi-fication to SMOTE for event classification in sodium cooled fast reactors (2017) Progr. Nucl. Energy, 100, pp. 355-364. , Sep; Mathew, J., Pang, C.K., Luo, M., Leong, W.H., Classification of imbalanced data by oversampling in kernel space of support vector machines (2018) IEEE Trans. Neural Netw. Learn. Syst., 29 (9), pp. 4065-4076. , Sep; Cheng, K., Zhang, C., Yu, H., Yang, X., Zou, H., Gao, S., Grouped SMOTE with noise filtering mechanism for classifying imbalanced data (2019) IEEE Access, 7, pp. 170668-170681. , Nov; He, H., Bai, Y., Garcia, E.A., Li, S., ADASYN: Adaptive synthetic sampling approach for imbalanced learning (2008) Proc. IEEE Int. Joint Conf. Neural Netw. (IEEE World Congr. Comput. Intell.), pp. 887-1322. , Jun; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Proc. 27th Int. Conf. Neural Inf. Process. Syst., pp. 2672-2680; Fiore, U., De Santis, A., Perla, F., Zanetti, P., Palmieri, F., Using generative adversarial networks for improving classification effectiveness in credit card fraud detection (2019) Inf. Sci., 479, pp. 448-455. , Apr; Douzas, G., Bacao, F., Effective data generation for imbalanced learning using conditional generative adversarial networks (2018) Expert Syst. Appl., 91, pp. 464-471. , Jan; Chawla, N.V., Lazarevic, A., Lawrence Hall, O., Bowyer, K., SMOTEBoost: Improving prediction of the minority class in boosting (2003) Proc. 7th Eur. Conf. Princ. Pract. Knowl. Discovery Database, pp. 107-119. , Jan; Joshi, M.V., Kumar, V., Agarwal, R.C., Evaluating boosting algorithms to classify rare classes: Comparison and improvements (2001) Proc. IEEE Int. Conf. Data Mining, pp. 257-264. , Dec; Tallo, T.E., Musdholifah, A., The implementation of genetic algorithm in SMOTE (Synthetic Minority Oversampling Technique) for handling imbalanced dataset problem (2018) Proc. 4th Int. Conf. Sci. Technol., pp. 1-4. , Aug; Ma, L., Fan, S., CURE-SMOTE algorithm and hybrid algorithm for feature selection and parameter optimization based on random forests (2017) BMC Bioinf., 18 (1). , Mar; Doroshenko, A., Piecewise-linear approach to classification based on geometrical transformation model for imbalanced dataset (2018) Proc. IEEE 2nd Int. Conf. Data Stream Mining Process. (DSMP), pp. 231-235. , Aug; Khan, S.H., Hayat, M., Bennamoun, M., Sohel, F.A., Togneri, R., Costsensitive learning of deep feature representations from imbalanced data (2018) IEEE Trans. Neural Netw. Learn. Syst., 29 (8), pp. 3573-3587. , Aug; Dhar, S., Cherkassky, V., Development and evaluation of cost-sensitive universum-SVM (2015) IEEE Trans. Cybern., 45 (4), pp. 806-818. , Apr; Qiu, C., Jiang, L., Li, C., Randomly selected decision tree for test-cost sensitive learning (2017) Appl. Soft Comput., 53, pp. 27-33. , Apr; Palacios, A., Trawiski, K., Cordón, O., Sánchez, L., Cost-sensitive learning of fuzzy rules for imbalanced classification problems using FURIA (2014) Int. J. Uncertainty, Fuzziness Knowl.-Based Syst., 22 (5), pp. 643-675. , Oct; Sun, Z., Song, Q., Zhu, X., Sun, H., Xu, B., Zhou, Y., A novel ensemble method for classifying imbalanced data (2015) Pattern Recognit., 48 (5), pp. 1623-1637. , May; Benchaji, I., Douzi, S., Elouahidi, B., Using genetic algorithm to improve classification of imbalanced data sets for credit card fraud detection (2018) Proc. 2nd Cyber Secur. Netw. Conf., pp. 1-5. , Oct; Awasare, V.K., Gupta, S., Classification of imbalanced data sets using partition method and support vector machine (2017) Proc. 2nd Int. Conf. Electr., Comput. Commun. Technol., pp. 1-7. , Feb; Barstufigan, M., Ceylan, R., A discriminative dictionary learning-AdaBoost-SVM classification method on imbalanced data sets (2017) Proc. Int. Artif. Intell. Data Process. Symp., pp. 1-4. , Sep; Landauer, T.K., Peter Foltz, W., Laham, D., An introduction to latent semantic analysis (2009) Discourse Process., 25 (2-3), pp. 259-284. , Nov; Mirza, M., Osindero, S., (2014) Conditional Generative Adversarial Nets, , http://arxiv.org/abs/1411.1784; Li, L., Zhang, H., Peng, H., Yang, Y., Nearest neighbors based density peaks approach to intrusion detection (2018) Chaos, Solitons Fractals, 110, pp. 33-40. , May; Yang, Y., Zheng, K., Wu, C., Niu, X., Yang, Y., Building an effective intrusion detection system using the modified density peak clustering algorithm and deep belief networks (2019) Appl. Sci., 9 (2), p. 238. , Jan; Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A., Improved training of wasserstein gans (2017) Proc. Neural Inf. Process. Syst., pp. 5767-5777. , Dec; Liu, A., Ghosh, J., Cheryl Martin, E., Generative oversampling for mining imbalanced data sets (2007) Proc. Int. Conf. Data Mining, pp. 66-72. , Jun; Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Duchesnay, É., Scikit-learn: Machine learning in Python (2011) J. Mach. Learn. Res., 12, pp. 2825-2830. , Oct","Wu, B.; School of Cyberspace Security, China; email: binwu@bupt.edu.cn",,,"Institute of Electrical and Electronics Engineers Inc.",,,,,21693536,,,,"English","IEEE Access",Article,"Final","All Open Access, Gold",Scopus,2-s2.0-85081131120
"Hayatbini N., Kong B., Hsu K.-L., Nguyen P., Sorooshian S., Stephens G., Fowlkes C., Nemani R., Ganguly S.","57195412150;57205353363;7401526171;55331455800;7005052907;57208765667;6701549596;35509463200;56012593900;","Conditional generative adversarial networks (cGANs) for near real-time precipitation estimation from multispectral GOES-16 satellite imageries-PERSIANN-cGAN",2019,"Remote Sensing","11","19","2193","","",,8,"10.3390/rs11192193","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85073408806&doi=10.3390%2frs11192193&partnerID=40&md5=ee563f287877925796524d0b1da5cda2","Center for Hydrometeorology and Remote Sensing (CHRS), The Henry Samueli School of Engineering, Department of Civil and Environmental Engineering, University of California, Irvine, CA  92697, United States; Department of Computer Sciences, University of California, Irvine, CA  92697, United States; Department of Earth System Science, University of California, Irvine, CA  92697, United States; Center for Climate Sciences, Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA  91109, United States; NASA Advanced Supercomputing Division, NASA Ames Research Center Moffet Field, Mountain View, CA  94035, United States; Bay Area Environmental Research Institute, NASA Ames Research Center, Moffett Field, CA  94035, United States","Hayatbini, N., Center for Hydrometeorology and Remote Sensing (CHRS), The Henry Samueli School of Engineering, Department of Civil and Environmental Engineering, University of California, Irvine, CA  92697, United States; Kong, B., Department of Computer Sciences, University of California, Irvine, CA  92697, United States; Hsu, K.-L., Center for Hydrometeorology and Remote Sensing (CHRS), The Henry Samueli School of Engineering, Department of Civil and Environmental Engineering, University of California, Irvine, CA  92697, United States; Nguyen, P., Center for Hydrometeorology and Remote Sensing (CHRS), The Henry Samueli School of Engineering, Department of Civil and Environmental Engineering, University of California, Irvine, CA  92697, United States; Sorooshian, S., Center for Hydrometeorology and Remote Sensing (CHRS), The Henry Samueli School of Engineering, Department of Civil and Environmental Engineering, University of California, Irvine, CA  92697, United States, Department of Earth System Science, University of California, Irvine, CA  92697, United States; Stephens, G., Center for Climate Sciences, Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA  91109, United States; Fowlkes, C., Department of Computer Sciences, University of California, Irvine, CA  92697, United States; Nemani, R., NASA Advanced Supercomputing Division, NASA Ames Research Center Moffet Field, Mountain View, CA  94035, United States; Ganguly, S., Bay Area Environmental Research Institute, NASA Ames Research Center, Moffett Field, CA  94035, United States","In this paper, we present a state-of-the-art precipitation estimation framework which leverages advances in satellite remote sensing as well as Deep Learning (DL). The framework takes advantage of the improvements in spatial, spectral and temporal resolutions of the Advanced Baseline Imager (ABI) onboard the GOES-16 platform along with elevation information to improve the precipitation estimates. The procedure begins by first deriving a Rain/No Rain (R/NR) binary mask through classification of the pixels and then applying regression to estimate the amount of rainfall for rainy pixels. A Fully Convolutional Network is used as a regressor to predict precipitation estimates. The network is trained using the non-saturating conditional Generative Adversarial Network (cGAN) and Mean Squared Error (MSE) loss terms to generate results that better learn the complex distribution of precipitation in the observed data. Common verification metrics such as Probability Of Detection (POD), False Alarm Ratio (FAR), Critical Success Index (CSI), Bias, Correlation and MSE are used to evaluate the accuracy of both R/NR classification and real-valued precipitation estimates. Statistics and visualizations of the evaluation measures show improvements in the precipitation retrieval accuracy in the proposed framework compared to the baseline models trained using conventional MSE loss terms. This framework is proposed as an augmentation for PERSIANN-CCS (Precipitation Estimation from Remotely Sensed Information using Artificial Neural Network- Cloud Classification System) algorithm for estimating global precipitation. © 2019 by the authors.","Convolutional neural networks (CNNs); Generative adversarial networks (GANs); Machine learning; Multispectral satellite imagery; Precipitation","Classification (of information); Convolution; Deep learning; Image enhancement; Learning algorithms; Learning systems; Mean square error; Neural networks; Pixels; Precipitation (chemical); Rain; Remote sensing; Adversarial networks; Cloud classification systems; Convolutional neural network; Multispectral satellite imagery; Precipitation estimation; Precipitation estimation from remotely sensed information; Precipitation retrievals; Probability of detection; Satellite imagery",,,,,"U.S. Department of Energy, USDOE: DE-IA0000018

National Aeronautics and Space Administration, NASA: NNX15AQ06A

California Energy Commission: 300-15-005

Jet Propulsion Laboratory, JPL: 1619578

Clinical Excellence Commission, CEC","The financial supports of this research are from U.S. Department of Energy (DOE Prime Award No. DE-IA0000018), California Energy Commission (CEC Award No. 300-15-005), MASEEH fellowship, NASA MIRO grant (NNX15AQ06A), and NASA-Jet Propulsion Laboratory (JPL) Grant (Award No. 1619578). The authors would like to thank the scientists at NASA Ames - Bay Area Environmental Research Institute (BAERI). Authors would also like to sincerely thank the valuable comments and suggestions of the editors and the anonymous reviewers.","Funding: The financial supports of this research are from U.S. Department of Energy (DOE Prime Award No. DE-IA0000018), California Energy Commission (CEC Award No. 300-15-005), MASEEH fellowship, NASA MIRO grant (NNX15AQ06A), and NASA—Jet Propulsion Laboratory (JPL) Grant (Award No. 1619578).","Sorooshian, S., AghaKouchak, A., Arkin, P., Eylander, J., Foufoula-Georgiou, E., Harmon, R., Hendrickx, J.M., Skahill, B., Advanced concepts on remote sensing of precipitation at multiple scales (2011) Bull. Am. Meteorol. Soc., 92, pp. 1353-1357; Nguyen, P., Shearer, E.J., Tran, H., Ombadi, M., Hayatbini, N., Palacios, T., Huynh, P., Hsu, K., The CHRS Data Portal, an easily accessible public repository for PERSIANN global satellite precipitation data (2019) Sci. Data, 6, p. 180296; Ba, M.B., Gruber, A., GOES multispectral rainfall algorithm (GMSRA) (2001) J. Appl. Meteorol., 40, pp. 1500-1514; Behrangi, A., Imam, B., Hsu, K., Sorooshian, S., Bellerby, T.J., Huffman, G.J., REFAME: Rain estimation using forward-adjusted advection of microwave estimates (2010) J. Hydrometeorol., 11, pp. 1305-1321; Behrangi, A., Hsu, K.L., Imam, B., Sorooshian, S., Huffman, G.J., Kuligowski, R.J., PERSIANN-MSA: A precipitation estimation method from satellite-based multispectral analysis (2009) J. Hydrometeorol., 10, pp. 1414-1429; Behrangi, A., Hsu, K.L., Imam, B., Sorooshian, S., Kuligowski, R.J., Evaluating the utility of multispectral information in delineating the areal extent of precipitation (2009) J. Hydrometeorol., 10, pp. 684-700; Martin, D.W., Kohrs, R.A., Mosher, F.R., Medaglia, C.M., Adamo, C., Over-ocean validation of the global convective diagnostic (2008) J. Appl. Meteorol. Climatol., 47, pp. 525-543; Tao, Y., Gao, X., Ihler, A., Hsu, K., Sorooshian, S., Deep neural networks for precipitation estimation from remotely sensed information (2016) Proceedings of the 2016 IEEE Congress on Evolutionary Computation (CEC), pp. 1349-1355. , Vancouver, BC, Canada, 24-29 July; Hayatbini, N., Hsu, K.L., Sorooshian, S., Zhang, Y., Zhang, F., Effective Cloud Detection and Segmentation Using a Gradient-Based Algorithm for Satellite Imagery: Application to Improve PERSIANN-CCS (2019) J. Hydrometeorol., 20, pp. 901-913; Joyce, R.J., Janowiak, J.E., Arkin, P.A., Xie, P., CMORPH: A method that produces global precipitation estimates from passive microwave and infrared data at high spatial and temporal resolution (2004) J. Hydrometeorol., 5, pp. 487-503; Kidd, C., Kniveton, D.R., Todd, M.C., Bellerby, T.J., Satellite rainfall estimation using combined passive microwave and infrared algorithms (2003) J. Hydrometeorol., 4, pp. 1088-1104; Huffman, G.J., Bolvin, D.T., Braithwaite, D., Hsu, K., Joyce, R., Xie, P., Yoo, S.H., NASA global precipitation measurement (GPM) integrated multi-satellite retrievals for GPM (IMERG) (2015) Algorithm Theor. Basis Doc. Version, 4, p. 30; Huffman, G.J., Bolvin, D.T., Nelkin, E.J., Wolff, D.B., Adler, R.F., Gu, G., Hong, Y., Stocker, E.F., The TRMM multisatellite precipitation analysis (TMPA): Quasi-global, multiyear, combined-sensor precipitation estimates at fine scales (2007) J. Hydrometeorol., 8, pp. 38-55; Hong, Y., Hsu, K.L., Sorooshian, S., Gao, X., Precipitation estimation from remotely sensed imagery using an artificial neural network cloud classification system (2004) J. Appl. Meteorol., 43, pp. 1834-1853; Tao, Y., Hsu, K., Ihler, A., Gao, X., Sorooshian, S., A two-stage deep neural network framework for precipitation estimation from Bispectral satellite information (2018) J. Hydrometeorol., 19, pp. 393-408; Bengio, Y., Learning deep architectures for AI (2009) Found. Trends Mach. Learn., 2, pp. 1-127; Hinton, G.E., Deep belief networks (2009) Scholarpedia, 4, p. 5947; LeCun, Y., Bengio, Y., Hinton, G., Deep learning (2015) Nature, 521, p. 436; Liu, Z., Zhou, P., Chen, X., Guan, Y., A multivariate conditional model for streamflow prediction and spatial precipitation refinement (2015) J. Geophys. Res. Atmos., p. 120; Rasp, S., Pritchard, M.S., Gentine, P., Deep learning to represent subgrid processes in climate models (2018) Proc. Natl. Acad. Sci. USA, 115, pp. 9684-9689; Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., Prabhat. Deep learning and process understanding for data-driven Earth system science (2019) Nature, 566, p. 195; Akbari Asanjan, A., Yang, T., Hsu, K., Sorooshian, S., Lin, J., Peng, Q., Short-Term Precipitation Forecast Based on the PERSIANN System and LSTM Recurrent Neural Networks (2018) J. Geophys. Res. Atmos., 123, pp. 12-543; Pan, B., Hsu, K., AghaKouchak, A., Sorooshian, S., Improving Precipitation Estimation Using Convolutional Neural Network (2019) Water Resour. Res., 55, pp. 2301-2321; Goodfellow, I., Bengio, Y., Courville, A., Deep Learning; MIT Press: Cambridge, MA, USA, 2016, , http://www.deeplearningbook.org, (accessed on 20 September 2019); Schmidhuber, J., Deep learning in neural networks: An overview (2015) Neural Netw., 61, pp. 85-117; Shen, D., Wu, G., Suk, H.I., Deep learning in medical image analysis (2017) Annu. Rev. Biomed. Eng., 19, pp. 221-248; Vandal, T., Kodra, E., Ganguly, A.R., Intercomparison of machine learning methods for statistical downscaling: The case of daily and extreme precipitation (2019) Theor. Appl. Climatol., 137, pp. 557-570; Tao, Y., Gao, X., Ihler, A., Sorooshian, S., Hsu, K., Precipitation identification with bispectral satellite information using deep learning approaches (2017) J. Hydrometeorol., 18, pp. 1271-1283; Liu, Y., Racah, E., Prabhat, C.J., Khosrowshahi, A., Lavers, D., Kunkel, K., Wehner, M., Collins, W., (2016) Application of deep convolutional neural networks for detecting extreme weather in climate datasets; Xingjian, S., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.C., Convolutional LSTM network: A machine learning approach for precipitation nowcasting (2015) Advances in Neural Information Processing Systems; The MIT Press: Cambridge, pp. 802-810. , MA, USA; LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D., Backpropagation applied to handwritten zip code recognition (1989) Neural Comput., 1, pp. 541-551; Elman, J.L., Finding structure in time (1990) Cogn. Sci., 14, pp. 179-211; Jordan, M.I., Serial order: A parallel distributed processing approach (1997) Advances in Psychology, 121, pp. 471-495. , Elsevier: Amsterdam, The Netherlands; Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks (2012) Advances in Neural Information Processing Systems; The MIT Press: Cambridge, pp. 1097-1105. , MA, USA; Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A., Extracting and composing robust features with denoising autoencoders (2008) Proceedings of the 25th International Conference on Machine Learning, pp. 1096-1103. , Helsinki, Finland, 5-9 July 2008; ACM: New York, NY, USA; Pu, Y., Gan, Z., Henao, R., Yuan, X., Li, C., Stevens, A., Carin, L., Variational autoencoder for deep learning of images, labels and captions (2016) Advances in Neural Information Processing Systems, pp. 2352-2360. , The MIT Press: Cambridge, MA, USA; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets. Advances in Neural Information Processing Systems (2014), pp. 2672-2680. , The MIT Press: Cambridge, MA, USA; Schmit, T.J., Gunshor, M.M., Menzel, W.P., Gurka, J.J., Li, J., Bachmeier, A.S., Introducing the next-generation Advanced Baseline Imager on GOES-R (2005) Bull. Am. Meteorol. Soc., 86, pp. 1079-1096; https://www.avl.class.noaa.gov/saa/products/welcome/, (accessed on 1 October 2018); Schmit, T.J., Menzel, W.P., Gurka, J., Gunshor, M., The ABI on GOES-R (2010) Proceedings of the 6th Annual Symposium on Future National Operational Environmental Satellite Systems-NPOESS and GOES-R, , Atlanta, GA, USA, 16-21 January; https://gpm-gv.gsfc.nasa.gov/, (accessed on 1 November 2018); Danielson, J.J., Gesch, D.B., (2011) Global Multi-Resolution Terrain Elevation Data 2010 (GMTED2010), Technical report, , US Geological Survey: Reston, VA, USA; Ronneberger, O., Fischer, P., Brox, T., U-net: Convolutional networks for biomedical image segmentation (2015) Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 5-9 October 2015, pp. 234-241. , Springer: Berlin, Germany; Arjovsky, M., Chintala, S., Bottou, L., (2017) Wasserstein gan; Huszár, F., (2015) How (not) to train your generative model: Scheduled sampling, likelihood, adversary?; Goodfellow, I., (2016) NIPS 2016 tutorial: Generative adversarial networks; Mirza, M., Osindero, S., (2014) Conditional generative adversarial nets; Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A., Image-to-image translation with conditional adversarial networks (2017) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125-1134. , Honolulu, HI, USA, 21-26 July","Hayatbini, N.; Center for Hydrometeorology and Remote Sensing (CHRS), United States; email: nhayatbi@uci.edu",,,"MDPI AG",,,,,20724292,,,,"English","Remote Sens.",Article,"Final","All Open Access, Gold",Scopus,2-s2.0-85073408806
"Dutta R., Raju S., James A., Leo C.J., Jeon Y.-J., Unnikrishnan B., Foo C.S., Zeng Z., Chai K.T.C., Chandrasekhar V.R.","56715015700;57197887058;12242134000;7006912505;57216898359;57210797719;14522280300;57194337704;36717471100;15064023000;","Learning of Multi-Dimensional Analog Circuits Through Generative Adversarial Network (GAN)",2019,"International System on Chip Conference","2019-September",,"9088077","394","399",,,"10.1109/SOCC46988.2019.1570548547","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85085202800&doi=10.1109%2fSOCC46988.2019.1570548547&partnerID=40&md5=dd298cb3416158fe310a79acf37f726d","Agency for Science, Technology and Research, ASTAR, Fusionopolis Way, Singapore, 138632, Singapore","Dutta, R., Agency for Science, Technology and Research, ASTAR, Fusionopolis Way, Singapore, 138632, Singapore; Raju, S., Agency for Science, Technology and Research, ASTAR, Fusionopolis Way, Singapore, 138632, Singapore; James, A., Agency for Science, Technology and Research, ASTAR, Fusionopolis Way, Singapore, 138632, Singapore; Leo, C.J., Agency for Science, Technology and Research, ASTAR, Fusionopolis Way, Singapore, 138632, Singapore; Jeon, Y.-J., Agency for Science, Technology and Research, ASTAR, Fusionopolis Way, Singapore, 138632, Singapore; Unnikrishnan, B., Agency for Science, Technology and Research, ASTAR, Fusionopolis Way, Singapore, 138632, Singapore; Foo, C.S., Agency for Science, Technology and Research, ASTAR, Fusionopolis Way, Singapore, 138632, Singapore; Zeng, Z., Agency for Science, Technology and Research, ASTAR, Fusionopolis Way, Singapore, 138632, Singapore; Chai, K.T.C., Agency for Science, Technology and Research, ASTAR, Fusionopolis Way, Singapore, 138632, Singapore; Chandrasekhar, V.R., Agency for Science, Technology and Research, ASTAR, Fusionopolis Way, Singapore, 138632, Singapore","Analog circuits are strictly designed under operational, functional and technology constraints. Together, these bounds create a sparse multi-dimensional design optimization space with the scarcity of labeled analog training data making supervised learning methods ineffective. Accurate approximation of multi-target analog circuits, therefore, requires generation of labeled data around dominant bias and with relevant variance. With such an approach, we explore state-of-the-art semi-supervised, generative adversarial network (GAN) towards analog performance modeling. We report on various multi-target analog circuit classification experiments and demonstrate stable GAN performance achieving 2-5% higher accuracy and utilizing only 10% fully simulated manually annotated labeled data against supervised learning methods. © 2019 IEEE.","Analog and mixed-signal circuits; generative adversarial network; machine learning; semi-supervised learning","Analog circuits; Labeled data; Programmable logic controllers; Supervised learning; Timing circuits; Adversarial networks; Analog performance; Circuit classification; Multi dimensional; Semi-supervised; State of the art; Supervised learning methods; Technology constraints; Learning systems",,,,,,,,"Takai, N., Fukuda, M., Prediction of element values of opamp for required specifications utilizing deep learning (2017) 2017 International Symposium on Electronics and Smart Devices (ISESD), pp. 300-303. , Yogyakarta; Lyu, W., Yang, F., Yan, C., Zhou, D., Zeng, X., Multi-objective Bayesian optimization for analog/rf circuit synthesis (2018) 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC)., pp. 1-6. , San Francisco, CA; Wang, F., Zhang, W., Sun, S., Li, X., Gu, C., Bayesian model fusion: Large-scale performance modeling of analog and mixed-signal circuits by reusing early-stage data (2013) 2013 50th ACM/EDAC/IEEE Design Automation Conference (DAC)., pp. 1-6. , Austin, TX; Liu, H., Singhee, A., Rutenbar, R.A., Carley, L.R., Remembrance of circuits past: Macromodeling by data mining in large analog design spaces (2002) Proceedings 2002 Design Automation Conference (IEEE Cat. No. 02CH37324), pp. 437-442. , New Orleans, LA, USA; Geman, S., Bienenstock, E., Doursat, R., Neural networks and the bias/variance dilemma (1992) Neural Computation, 4 (1), pp. 1-58. , Jan; Parsian, M., (2012) Data Algorithms Recipes for Scaling Up with Hadoop and Spark, , Pearson's Correlation Coefficient, O'reilly Publications; Bickel, S., Brückner, M., Scheffer, T., Discriminative learning under covariate shift (2009) J. Mach. Learn. Res., 10, pp. 2137-2155. , December; De Bernardinis, F., Jordan, M.I., SangiovanniVincentelli, A., Support vector machines for analog circuit performance representation (2003) Proceedings 2003. Design Automation Conference (IEEE Cat. No. 03CH37451), pp. 964-969. , Anaheim, CA; ImageNet: A Large-scale Hierarchical Image Database, , http://vision.stanford.edu/pdf/ImageNet-CVPR2009.pdf, Link; Tuyls, K., Weiss, G., Multiagent learning: Basics, challenges, and prospects (2012) AIMag, 33 (3), p. 41. , Sep; Belkin, M., Niyogi, P., Sindhwani, V., Manifold regularization: A geometric framework for learning from labeled and unlabeled examples (2006) J. Mach. Learn. Res., 7, pp. 2399-2434. , December; Kumar, A., Sattigeri, P., Fletcher, P.T., Semisupervised learning with gans: Manifold invariance with improved inference Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17, pp. 5540-5550; Belkin, M., Niyogi, P., Semi-supervised learning on riemannian manifolds (2004) Mach. Learn., 56 (1-3), pp. 209-239. , June; Miyato, T., Maeda, S., Koyama, M., Ishii, S., Virtual adversarial training: A regularization method for supervised and semi-supervised learning (2019) IEEE Transactions on Pattern Analysis and Machine Intelligence, 41 (8), pp. 1979-1993. , 1 Aug; Simard, P., Victorri, B., Le Cun, Y., Denker, J., Tangent prop: A formalism for specifying selected invariances in an adaptive network Proceedings of the 4th International Conference on Neural Information Processing Systems (NIPS'91, pp. 895-903; Bruno, L., Chuan-Sheng, F., Houssam, Z., Vijay, C., (2018) Manifold Regularization with GANs for Semi-supervised Learning, , https://arxiv.org/abs/1807.04307, arXiv:1807.04307, Link",,"Zhao D.Basu A.Bayoumi M.Hwee G.B.Tong G.Sridhar R.","IEEE Circuits and Systems Society","IEEE Computer Society","32nd IEEE International System on Chip Conference, SOCC 2019","3 September 2019 through 6 September 2019",,159733,21641676,9781728134826,,,"English","Int. Syst. Chip Conf.",Conference Paper,"Final","",Scopus,2-s2.0-85085202800
"Armanious K., Küstner T., Reimold M., Nikolaou K., La Fougère C., Yang B., Gatidis S.","57208782510;56549927000;6508312821;55744104000;10642092800;55584795030;26658018800;","Independent brain 18F-FDG PET attenuation correction using a deep learning approach with Generative Adversarial Networks",2019,"Hellenic journal of nuclear medicine","22","3",,"179","186",,13,"10.1967/s002449911053","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85074184660&doi=10.1967%2fs002449911053&partnerID=40&md5=631789add1c44449a0d916fc53d43df5","University Hospital Tübingen, Department of Radiology, Diagnostic and Interventional Radiology, Tübingen, University of Stuttgart, Institute of Signal Processing and System Theory, Stuttgart, Germany","Armanious, K., University Hospital Tübingen, Department of Radiology, Diagnostic and Interventional Radiology, Tübingen, University of Stuttgart, Institute of Signal Processing and System Theory, Stuttgart, Germany; Küstner, T.; Reimold, M.; Nikolaou, K.; La Fougère, C.; Yang, B.; Gatidis, S.","OBJECTIVE: Attenuation correction (AC) of positron emission tomography (PET) data poses a challenge when no transmission data or computed tomography (CT) data are available, e.g. in stand alone PET scanners or PET/magnetic resonance imaging (MRI). In these cases, external imaging data or morphological imaging data are normally used for the generation of attenuation maps. Newly introduced machine learning methods however may allow for direct estimation of attenuation maps from non attenuation-corrected PET data (PETNAC). Our purpose was thus to establish and evaluate a method for independent AC of brain fluorine-18-fluorodeoxyglucose (18F-FDG) PET images only based on PETNAC using Generative Adversarial Networks (GAN). SUBJECTS AND METHODS: After training of the deep learning GAN framework on a paired training dataset of PETNAC and the corresponding CT images of the head from 50 patients, pseudo-CT images were generated from PETNAC of 40 validation patients, of which 20 were used for technical validation and 20 stemming from patients with CNS disorders were used for clinical validation. Pseudo-CT was used for subsequent AC of these validation data sets resulting in independently attenuation-corrected PET data. RESULTS: Visual inspection revealed a high degree of resemblance of generated pseudo-CT images compared to the acquired CT images in all validation data sets, with minor differences in individual anatomical details. Quantitative analyses revealed minimal underestimation below 5% of standardized uptake value (SUV) in all brain regions in independently attenuation-corrected PET data compared to the reference PET images. Color-coded error maps showed no regional bias and only minimal average errors around ±0%. Using independently attenuation-corrected PET data, no differences in image-based diagnoses were observed in 20 patients with neurological disorders compared to the reference PET images. CONCLUSION: Independent AC of brain 18F-FDG PET is feasible with high accuracy using the proposed, easy to implement deep learning framework. Further evaluation in clinical cohorts will be necessary to assess the clinical performance of this method.",,"fluorodeoxyglucose f 18; brain; diagnostic imaging; human; image processing; male; middle aged; positron emission tomography-computed tomography; procedures; Brain; Deep Learning; Fluorodeoxyglucose F18; Humans; Image Processing, Computer-Assisted; Male; Middle Aged; Positron Emission Tomography Computed Tomography",,"fluorodeoxyglucose f 18, 63503-12-8; Fluorodeoxyglucose F18",,,,,,,,,,"NLM (Medline)",,,,,17905427,,,"31587027","English","Hell J Nucl Med",Article,"Final","",Scopus,2-s2.0-85074184660
"Olmschenk G., Zhu Z., Tang H.","55360998300;7404803571;57194711317;","Generalizing semi-supervised generative adversarial networks to regression using feature contrasting",2019,"Computer Vision and Image Understanding","186",,,"1","12",,7,"10.1016/j.cviu.2019.06.004","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85068990139&doi=10.1016%2fj.cviu.2019.06.004&partnerID=40&md5=6b0b73326e9cbd468e757c0a8c9f60c8","The City College, The City University of New York, 160 Convent Ave, New York, NY  10031, United States; The Graduate Center, The City University of New York, 365 5th Ave, New York, NY  10016, United States; Borough of Manhattan Community College, The City University of New York, 199 Chambers St, New York, NY  10007, United States","Olmschenk, G., The City College, The City University of New York, 160 Convent Ave, New York, NY  10031, United States, The Graduate Center, The City University of New York, 365 5th Ave, New York, NY  10016, United States; Zhu, Z., The City College, The City University of New York, 160 Convent Ave, New York, NY  10031, United States, The Graduate Center, The City University of New York, 365 5th Ave, New York, NY  10016, United States; Tang, H., Borough of Manhattan Community College, The City University of New York, 199 Chambers St, New York, NY  10007, United States","In this work, we generalize semi-supervised generative adversarial networks (GANs) from classification problems to regression problems. In the last few years, the importance of improving the training of neural networks using semi-supervised training has been demonstrated for classification problems. We present a novel loss function, called feature contrasting, resulting in a discriminator which can distinguish between fake and real data based on feature statistics. This method avoids potential biases and limitations of alternative approaches. The generalization of semi-supervised GANs to the regime of regression problems of opens their use to countless applications as well as providing an avenue for a deeper understanding of how GANs function. We first demonstrate the capabilities of semi-supervised regression GANs on a toy dataset which allows for a detailed understanding of how they operate in various circumstances. This toy dataset is used to provide a theoretical basis of the semi-supervised regression GAN. We then apply the semi-supervised regression GANs to a number of real-world computer vision applications: age estimation, driving steering angle prediction, and crowd counting from single images. We perform extensive tests of what accuracy can be achieved with significantly reduced annotated data. Through the combination of the theoretical example and real-world scenarios, we demonstrate how semi-supervised GANs can be generalized to regression problems. © 2019 Elsevier Inc.","Age estimation; Generative adversarial learning; Regression","Image understanding; Software engineering; Adversarial learning; Adversarial networks; Age estimation; Computer vision applications; Real-world scenario; Regression; Regression problem; Semi-supervised trainings; Regression analysis",,,,,"National Science Foundation, NSF: 1137172, 1737533, 1827505

U.S. Department of Energy, USDOE

U.S. Department of Homeland Security, DHS

Department of Human Services, DHS

Oak Ridge Associated Universities, ORAU: 1 DE-AC05-06OR23100, 1 DE-SC0014664

Oak Ridge Institute for Science and Education, ORISE

Defense Intelligence Agency, DIA","This research was initiated under appointments to the U.S. Department of Homeland Security (DHS) Science & Technology Directorate Office of University Programs, administered by the Oak Ridge Institute for Science and Education (ORISE) through an interagency agreement between the U.S. Department of Energy (DOE) and DHS. ORISE is managed by ORAU under DOE contract number 1 DE-AC05-06OR23100 and 1 DE-SC0014664. All opinions expressed in this paper are the author’s and do not necessarily reflect the policies and views of DHS, DOE, or ORAU/ORISE. The research is also supported by National Science Foundation through Awards PFI #1827505 and SCC-Planning #1737533 , and Bentley Systems, Incorporated , through a CUNY-Bentley Collaborative Research Agreement (CRA). Additional support provided by the Defense Intelligence Agency (DIA) via the Rutgers University Consortium for Critical Technology Studies.","This research was initiated under appointments to the U.S. Department of Homeland Security (DHS) Science & Technology Directorate Office of University Programs, administered by the Oak Ridge Institute for Science and Education (ORISE) through an interagency agreement between the U.S. Department of Energy (DOE) and DHS. ORISE is managed by ORAU under DOE contract number 1DE-AC05-06OR23100 and 1DE-SC0014664. All opinions expressed in this paper are the author's and do not necessarily reflect the policies and views of DHS, DOE, or ORAU/ORISE. The research is also supported by National Science Foundation through Awards PFI #1827505 and SCC-Planning #1737533, and Bentley Systems, Incorporated, through a CUNY-Bentley Collaborative Research Agreement (CRA). Additional support provided by the Defense Intelligence Agency (DIA) via the Rutgers University Consortium for Critical Technology Studies.","Ali, I., Greifeneder, F., Stamenkovic, J., Neumann, M., Notarnicola, C., Review of machine learning approaches for biomass and soil moisture retrievals from remote sensing data (2015) Remote Sens., 7 (12), pp. 16398-16421; Arjovsky, M., Chintala, S., Bottou, L., Wasserstein gan, arXiv preprint (2017); (2018), Barnett, S.A. Convergence Problems with Generative Adversarial Networks (GANs), arXiv preprint; Bazrafkan, S., Corcoran, P., Versatile Auxiliary Regressor with Generative Adversarial network (VAR+ GAN), arXiv preprint (2018); Bland, L.M., Collen, B., Orme, C.L., Bielby, J., Predicting the conservation status of data-deficient species (2015) Conserv. Biol., 29 (1), pp. 250-259; Chen, S., (2017), Sully Chen Driving Dataset. [Online; accessed 8-February-2019]; Dai, Z., Yang, Z., Yang, F., Cohen, W.W., Salakhutdinov, R.R., Good semi-supervised learning that requires a bad gan (2017) Advances in Neural Information Processing Systems, pp. 6513-6523; Ding, X., Zhang, Y., Liu, T., Duan, J., Deep learning for event-driven stock prediction. (2015) Ijcai, pp. 2327-2333; Dodge, S., Karam, L., A Study and Comparison of Human and Deep Learning Recognition Performance Under Visual Distortions, arXiv preprint (2017); Eigen, D., Puhrsch, C., Fergus, R., Depth map prediction from a single image using a multi-scale deep network (2014) Advances in Neural Information Processing Systems, pp. 2366-2374; Elsayed, G., Shankar, S., Cheung, B., Papernot, N., Kurakin, A., Goodfellow, I., Sohl-Dickstein, J., Adversarial examples that fool both computer vision and time-limited humans (2018) Advances in Neural Information Processing Systems, pp. 3914-3924; Fabbro, S., Venn, K., O'Briain, T., Bialek, S., Kielty, C., Jahandar, F., Monty, S., An application of deep learning in the analysis of stellar spectra (2017) Mon. Not. R. Astron. Soc.; Fefferman, C., Mitter, S., Narayanan, H., Testing the manifold hypothesis (2016) J. Amer. Math. Soc., 29 (4), pp. 983-1049; Goodfellow, I., NIPS 2016 tutorial: Generative adversarial networks, arXiv preprint (2016); Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Advances in Neural Information Processing Systems, pp. 2672-2680; Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C., Improved training of wasserstein gans (2017) Advances in Neural Information Processing Systems, pp. 5769-5779; Hartikainen, J., Seppanen, M., Sarkka, S., State-space inference for non-linear latent force models with application to satellite orbit prediction, arXiv preprint (2012); Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q., Densely connected convolutional networks. (2017) CVPR, Vol. 1, p. 3; Idrees, H., Tayyab, M., Athrey, K., Zhang, D., Al-Maadeed, S., Rajpoot, N., Shah, M., Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds, arXiv preprint (2018); LeCun, Y., Bengio, Y., Hinton, G., Deep learning (2015) Nature, 521 (7553), pp. 436-444; LeCun, Y., Haffner, P., Bottou, L., Bengio, Y., (1999), Object recognition with gradient-based learning, Shape, contour and grouping in computer vision, 823–823; Liu, Y., Weisberg, R.H., Patterns of ocean current variability on the west florida shelf using the self-organizing map (2005) J. Geophys. Res.: Oceans, 110 (C6); Lv, Y., Duan, Y., Kang, W., Li, Z., Wang, F.-Y., Traffic flow prediction with big data: a deep learning approach (2015) IEEE Trans. Intell. Transp. Syst., 16 (2), pp. 865-873; Marino, D.L., Amarasinghe, K., Manic, M., Building energy load forecasting using deep neural networks (2016) Industrial Electronics Society, IECON 2016-42nd Annual Conference of the IEEE, pp. 7046-7051. , IEEE; Niu, Z., Zhou, M., Wang, L., Gao, X., Hua, G., Ordinal regression with multiple output cnn for age estimation (2016) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4920-4928; Oliveira, T.P., Barbar, J.S., Soares, A.S., Computer network traffic prediction: a comparison between traditional and deep learning neural networks (2016) Int. J. Big Data Intell., 3 (1), pp. 28-37; Pan, X., You, Y., Wang, Z., Lu, C., Virtual to real reinforcement learning for autonomous driving, arXiv preprint (2017); Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A., Context encoders: Feature learning by inpainting (2016) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536-2544; Radford, A., Metz, L., Chintala, S., Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint (2015); Rezagholiradeh, M., Haidar, M.A., Reg-gan: Semi-supervised learning based on generative adversarial networks for regression (2018) 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2806-2810. , IEEE; Rothe, R., Timofte, R., Van Gool, L., Deep expectation of real and apparent age from a single image without facial landmarks (2016) Int. J. Comput. Vis., pp. 1-14; Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X., Improved techniques for training gans (2016) Advances in Neural Information Processing Systems, pp. 2234-2242; Schwarz, M., Schulz, H., Behnke, S., RGB-D object recognition and pose estimation based on pre-trained convolutional neural network features (2015) Robotics and Automation (ICRA), 2015 IEEE International Conference on, pp. 1329-1335. , IEEE; Souly, N., Spampinato, C., Shah, M., Semi and Weakly Supervised Semantic Segmentation Using Generative Adversarial Network, arXiv preprint (2017); (2015), Springenberg, J.T. Unsupervised and semi-supervised learning with categorical generative adversarial networks, arXiv preprint; Sricharan, K., Bala, R., Shreve, M., Ding, H., Saketh, K., Sun, J., Semi-supervised conditional gans, arXiv preprint (2017); Xingjian, S., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., Woo, W.-C., Convolutional LSTM network: A machine learning approach for precipitation nowcasting (2015) Advances in Neural Information Processing Systems, pp. 802-810; Zhang, C., Li, H., Wang, X., Yang, X., Cross-scene crowd counting via deep convolutional neural networks (2015) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 833-841; Zhang, Y., Zhou, D., Chen, S., Gao, S., Ma, Y., Single-image crowd counting via multi-column convolutional neural network (2016) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 589-597","Olmschenk, G.; The Graduate Center, 365 5th Ave, United States; email: golmschenk@gradcenter.cuny.edu",,,"Academic Press Inc.",,,,,10773142,,CVIUF,,"English","Comput Vision Image Understanding",Article,"Final","All Open Access, Green",Scopus,2-s2.0-85068990139
"Sattigeri P., Hoffman S.C., Chenthamarakshan V., Varshney K.R.","35180457200;57209497480;36847907800;14069407800;","Fairness GAN: Generating datasets with fairness properties using a generative adversarial network",2019,"IBM Journal of Research and Development","63","4-5","8869910","","",,12,"10.1147/JRD.2019.2945519","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85075099807&doi=10.1147%2fJRD.2019.2945519&partnerID=40&md5=3e21bf8b0c3916e84ba50a1ca3d528bb","IBM Research, Yorktown Heights, NY  10598, United States","Sattigeri, P., IBM Research, Yorktown Heights, NY  10598, United States; Hoffman, S.C., IBM Research, Yorktown Heights, NY  10598, United States; Chenthamarakshan, V., IBM Research, Yorktown Heights, NY  10598, United States; Varshney, K.R., IBM Research, Yorktown Heights, NY  10598, United States","We introduce the Fairness GAN (generative adversarial network), an approach for generating a dataset that is plausibly similar to a given multimedia dataset, but is more fair with respect to protected attributes in decision making. We propose a novel auxiliary classifier GAN that strives for demographic parity or equality of opportunity and show empirical results on several datasets, including the CelebFaces Attributes (CelebA) dataset, the Quick, Draw! dataset, and a dataset of soccer player images and the offenses for which they were called. The proposed formulation is well suited to absorbing unlabeled data; we leverage this to augment the soccer dataset with the much larger CelebA dataset. The methodology tends to improve demographic parity and equality of opportunity while generating plausible images. © 1957-2012 IBM.",,"Decision making; Image enhancement; Population statistics; Sports; Adversarial networks; Fairness properties; Soccer player; Unlabeled data; Classification (of information)",,,,,,,,"Shermis, M.D., State-of-The-Art automated essay scoring: Competition, results, and future directions from a United States demonstration (2014) Assessing Writing, 20, pp. 53-76. , Apr; Perelman, L., When 'the state of the art' is counting words (2014) Assessing Writing, 21, pp. 104-111. , Jul; Shahani, A., (2015) Now Algorithms Are Deciding Whom to Hire, Based on Voice, , https://www.npr.org/sections/alltechconsidered/2015/03/23/394827451/now-algorithms-aredeciding-whom-to-hire-based-on-voice; Chandler, S., (2017) The AI Chatbot Will Hire You Now, , https://www.wired.com/story/the-ai-chatbot-will-hireyou-now/; Williams, B.A., Brooks, C.F., Shmargad, Y., How algorithms discriminate based on data they lack: Challenges, solutions, and policy implications (2018) J. Inf. Policy, 8, pp. 78-115; D'Alessandro, B., O'Neil, C., LaGatta, T., Conscientious classification: A data scientist's guide to discrimination-aware classification (2017) Big Data, 5 (2), pp. 120-134; Kamiran, F., Calders, T., Data preprocessing techniques for classification without discrimination (2012) Knowl. Inf. Syst., 33 (1), pp. 1-33; Goodfellow, I., (2017) NIPS 2016 Tutorial: Generative Adversarial Networks, , arXiv:1701.00160; Odena, A., Olah, C., Shlens, J., Conditional image synthesis with auxiliary classifier GANs (2017) Proc. Int. Conf. Mach. Learn., pp. 2642-2651; Edwards, H., Storkey, A., Censoring representations with an adversary (2016) Proc. Int. Conf. Learn. Rep.; Beutel, A., Chen, J., Zhao, Z., Data decisions and theoretical implications when adversarially learning fair representations (2017) Proc. Workshop Fairness, Accountability, Transparency Mach. Learn.; Zhang, B.H., Lemoine, B., Mitchell, M., Mitigating unwanted biases with adversarial learning (2018) Proc. AAAI/ACM Conf. Artif. Intell., Ethics, Soc., pp. 335-340; Madras, D., Creager, E., Pitassi, T., Learning adversarially fair and transferable representations (2018) Proc. Int. Conf. Mach. Learn., pp. 3384-3393. , July; Wadsworth, C., Vera, F., Piech, C., Achieving fairness through adversarial learning: An application to recidivism prediction (2018) Proc. Workshop Fairness, Accountability, Transparency Mach. Learn.; Adel, T., Valera, I., Ghahramani, Z., One-network adversarial fairness (2019) Proc. 33rd AAAI Conf. Artif. Intell., pp. 2412-2420; Elazar, Y., Goldberg, Y., Adversarial removal of demographic attributes from text data (2018) Proc. Conf. Empirical Methods Natural Lang. Process., pp. 11-21; Perarnau, G., Van De Weijer, J., Raducanu, B., Invertible conditional GANs for image editing (2016) Proc. NIPS Workshop Adversarial Training; Zhu, J.-Y., Park, T., Isola, P., Unpaired image-to-image translation using cycle-consistent adversarial networks (2017) Proc. IEEE Int. Conf. Comput. Vis., pp. 2242-2251; Lu, Y., Tai, Y.-W., Tang, C.-K., (2017) Conditional CycleGAN for Attribute Guided Face Image Generation, , arXiv:1705.09966; Liu, Z., Luo, P., Wang, X., Deep learning face attributes in the wild (2015) Proc. IEEE Int. Conf. Comput. Vis., pp. 3730-3738; Silberzahn, R., Uhlmann, E.L., Martin, D.P., Many analysts, one dataset: Making transparent how variations in analytical choices affect results (2017) Adv. Methods Pract. Psychol. Sci., 1 (3), pp. 337-356; Dumoulin, V., Shlens, J., Kudlur, M., A learned representation for artistic style (2017) Proc. Int. Conf. Learn. Rep.; Miyato, T., Koyama, M., CGANs with projection discriminator (2018) Proc. Int. Conf. Learn. Rep.; Miyato, T., Kataoka, T., Koyama, M., Spectral normalization for generative adversarial networks (2018) Proc. Int. Conf. Learn. Rep.; Lim, J.H., Ye, J.C., (2017) Geometric GAN, , arXiv:1705.02894; Xu, D., Yuan, S., Zhang, L., FairGAN: Fairness-aware generative adversarial networks (2018) Proc. IEEE Int. Conf. Big Data, pp. 570-575. , Dec; Böhlen, M., Chandola, V., Salunkhe, A., (2017) Server, Server in the Cloud. Who Is the Fairest in the Crowd?, , arXiv:1711.08801; (2016) Quickdraw with Google Didn't Recognise My Drawing of A Power Outlet because i Drew A 3-pinned Rectangular Plug, , https://www.reddit.com/r/britishproblems/comments/5deygp/quickdraw-withgoogle-didnt-recognise-my-drawing, TailSpectrum; Gulrajani, I., Ahmed, F., Arjovsky, M., Improved training of wasserstein GANs (2017) Proc. Adv. Neural Inf. Process. Syst., pp. 5769-5779; He, K., Zhang, X., Ren, S., Deep residual learning for image recognition (2016) Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 770-778; Jang, E., Gu, S., Poole, B., Categorical reparameterization with gumbel-softmax (2017) Proc. Int. Conf. Learn. Rep.; Maddison, C.J., Mnih, A., Teh, Y.W., The concrete distribution: A continuous relaxation of discrete random variables (2017) Pro. Int. Conf. Learn. Rep.; Turk, M., Pentland, A., Eigenfaces for recognition (1991) J. Cogn. Neurosci., 3 (1), pp. 71-86; Friedler, S.A., Scheidegger, C., Venkatasubramanian, S., A comparative study of fairness-enhancing interventions in machine learning (2019) Proc. ACM Conf. Fairness, Accountability Transparency, pp. 329-338. , Jan; Louppe, G., Kagan, M., Cranmer, K., Learning to pivot with adversarial networks (2017) Proc. Adv. Neural Inf. Process. Syst., pp. 981-990",,,,"IBM Corporation",,,,,00188646,,IBMJA,,"English","IBM J. Res. Dev.",Article,"Final","",Scopus,2-s2.0-85075099807
"Olmschenk G., Chen J., Tang H., Zhu Z.","55360998300;57219310771;57194711317;7404803571;","Dense crowd counting convolutional neural networks with minimal data using semi-supervised dual-goal generative adversarial networks",2019,"IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops","2019-June",,,"21","28",,3,,"https://www.scopus.com/inward/record.uri?eid=2-s2.0-85095357088&partnerID=40&md5=e0b59959d5809a02e58d96d57f192886","The Graduate Center of the City University of New York, United States; The City College of New York - CUNY, United States; Borough of Manhattan Community College - CUNY","Olmschenk, G., The Graduate Center of the City University of New York, United States; Chen, J., The City College of New York - CUNY, United States; Tang, H., Borough of Manhattan Community College - CUNY; Zhu, Z., The Graduate Center of the City University of New York, United States, The City College of New York - CUNY, United States","In this work, we generalize semi-supervised generative adversarial networks (GANs) from classification problems to regression for use in dense crowd counting. In the last several years, the importance of improving the training of neural networks using semi-supervised training has been thoroughly demonstrated for classification problems. This work presents a dual-goal GAN which seeks both to provide the number of individuals in a densely crowded scene and distinguish between real and generated images. This method allows the dual-goal GAN to benefit from unlabeled data in the training process, improving the predictive capabilities of the discriminating network compared to the fully-supervised version of the network. Typical semi-supervised GANs are unable to function in the regression regime due to biases introduced when using a single prediction goal. Using the proposed approach, the amount of data which needs to be annotated for dense crowd counting can be significantly reduced. © 2019 IEEE Computer Society. All rights reserved.",,"Computer vision; Adversarial networks; Predictive capabilities; Semi-supervised; Semi-supervised trainings; Training process; Unlabeled data; Convolutional neural networks",,,,,"National Science Foundation, NSF: -1737533, -1827505","This research is supported by the National Science Foundation via awards #CNS-1737533 and #IIP-1827505, and by Bentley Systems, Inc., through a CUNY-Bentley Collaborative Research Agreement (CRA).",,"Badrinarayanan, V., Kendall, A., SegNet, R.C., A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation, , arXiv preprint; Fefferman, C., Mitter, S., Narayanan, H., Testing the manifold hypothesis (2016) Journal of the American Mathematical Society, 29 (4), pp. 983-1049. , 2; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Advances in Neural Information Processing Systems, pp. 2672-2680. , 2; Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q., Densely connected convolutional networks (2017) CVPR, 1, p. 3. , 5; Idrees, H., Saleemi, I., Seibert, C., Shah, M., Multi-source multi-scale counting in extremely dense crowd images (2013) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2547-2554. , 8; Idrees, H., Tayyab, M., Athrey, K., Zhang, D., Al-Maadeed, S., Rajpoot, N., Shah, M., (2018) Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds, , arXiv preprint 5, 6; Radford, A., Metz, L., Chintala, S., (2015) Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, , arXiv preprint 3, 5; Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X., Improved techniques for training gans (2016) Advances in Neural Information Processing Systems, pp. 2234-2242. , 3, 4; Sam, D.B., Surya, S., Babu, R.V., Switching convolutional neural network for crowd counting (2017) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1, p. 6. , 5; Sindagi, V.A., Patel, V.M., Cnn-based cascaded multi-task learning of high-level prior and density estimation for crowd counting (2017) Advanced Video and Signal Based Surveillance (AVSS), 2017 14th IEEE International Conference on, pp. 1-6. , IEEE, 5; Souly, N., Spampinato, C., Shah, M., (2017) Semi and Weakly Supervised Semantic Segmentation Using Generative Adversarial Network, , arXiv preprint 4; Springenberg, J.T., (2015) Unsupervised and Semi-Supervised Learning with Categorical Generative Adversarial Networks, , arXiv preprint 4; Sricharan, K., Bala, R., Shreve, M., Ding, H., Saketh, K., Sun, J., (2017) Semi-Supervised Conditional Gans, , arXiv preprint 4; Zhang, C., Li, H., Wang, X., Yang, X., Cross-scene crowd counting via deep convolutional neural networks (2015) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 833-841. , 8; Zhang, Y., Zhou, D., Chen, S., Gao, S., Ma, Y., Single-image crowd counting via multi-column convolutional neural network (2016) Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 589-597. , 5, 8",,,,"IEEE Computer Society","32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2019","16 June 2019 through 20 June 2019",,159074,21607508,9781728125060,,,"English","IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn. Workshops",Conference Paper,"Final","",Scopus,2-s2.0-85095357088
"Xu D., Yuan S., Zhang L., Wu X.","57201947036;56486630300;57191309823;8426129400;","FairGAN: Fairness-aware Generative Adversarial Networks",2019,"Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018",,,"8622525","570","575",,27,"10.1109/BigData.2018.8622525","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85062597161&doi=10.1109%2fBigData.2018.8622525&partnerID=40&md5=5fcdee92df39f5d5162308efb23a1928","University of Arkansas, Fayetteville, AR, United States","Xu, D., University of Arkansas, Fayetteville, AR, United States; Yuan, S., University of Arkansas, Fayetteville, AR, United States; Zhang, L., University of Arkansas, Fayetteville, AR, United States; Wu, X., University of Arkansas, Fayetteville, AR, United States","Fairness-aware learning is increasingly important in data mining. Discrimination prevention aims to prevent discrimination in the training data before it is used to conduct predictive analysis. In this paper, we focus on fair data generation that ensures the generated data is discrimination free. Inspired by generative adversarial networks (GAN), we present fairness-aware generative adversarial networks, called FairGAN, which are able to learn a generator producing fair data and also preserving good data utility. Compared with the naive fair data generation models, FairGAN further ensures the classifiers which are trained on generated data can achieve fair classification on real data. Experiments on a real dataset show the effectiveness of FairGAN. © 2018 IEEE.",,"Big data; Data mining; Adversarial networks; Data generation; Data generation models; Good data; Training data; Classification (of information)",,,,,"National Science Foundation, NSF: 1564250, 1646654, 1841119","ACKNOWLEDGMENT This work was supported in part by NSF 1564250, 1646654 and 1841119.","This work was supported in part by NSF 1564250, 1646654 and 1841119.","Joseph, M., Kearns, M., Morgenstern, J.H., Roth, A., Fairness in learning: Classic and contextual bandits (2016) NIPS; Beutel, A., Chen, J., Zhao, Z., Chi, E.H., Data decisions and theoretical implications when adversarially learning fair representations (2017) FAT/ML; Binns, R., (2017) Fairness in Machine Learning: Lessons from Political Philosophy; Zhang, L., Wu, Y., Wu, X., A causal framework for discovering and removing direct and indirect discrimination (2017) IJCAI; Kamiran, F., Calders, T., Classifying without discriminating (2009) Control and Communication 2009 2nd International Conference on Computer; Data preprocessing techniques for classification without discrimination (2012) Knowledge and Information Systems, 33 (1), pp. 1-33; Feldman, M., Friedler, S., Moeller, J., Scheidegger, C., Venkatasubramanian, S., Certifying and removing disparate impact (2015) KDD; Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Ozair, S., Courville, A., Bengio, Y., Generative adversarial networks (2014) NIPS; Radford, A., Metz, L., Chintala, S., (2015) Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks; Choi, E., Biswal, S., Malin, B., Duke, J., Stewart, W.F., Sun, J., Generating multi-label discrete patient records using generative adversarial networks (2017) MLHC; Calders, T., Kamiran, F., Pechenizkiy, M., Building classifiers with independency constraints (2009) ICDM Workshops; Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R., (2011) Fairness Through Awareness; Wu, Y., Wu, X., Using loglinear model for discrimination discovery and prevention (2016) DSAA; Kamishima, T., Akaho, S., Sakuma, J., Fairness-aware learning through regularization approach (2011) ICDM Workshops; Zafar, M.B., Valera, I., Rodriguez, M.G., Gummadi, K.P., Fairness constraints: Mechanisms for fair classification (2017) AISTATS; Kamiran, F., Calders, T., Pechenizkiy, M., Discrimination aware decision tree learning (2010) ICDM; Hardt, M., Price, E., Srebro, N., Equality of opportunity in supervised learning (2016) NIPS; Zhang, L., Wu, Y., Wu, X., Achieving non-discrimination in data release (2017) KDD; Edwards, H., Storkey, A., (2015) Censoring Representations with an Adversary; Madras, D., Creager, E., Pitassi, T., Zemel, R., (2018) Learning Adversarially Fair and Transferable Representations; Zhang, B.H., Lemoine, B., Mitchell, M., Mitigating unwanted biases with adversarial learning (2018) AIES; Mirza, M., Osindero, S., (2014) Conditional Generative Adversarial Nets; Dheeru, D., Karra, E., (2017) Taniskidou UCI Machine Learning Repository, , University of California, Irvine, School of Information and Computer Sciences; Kingma, D.P., Ba, J., Adam: A method for stochastic optimization (2015) ICLR",,"Song Y.Liu B.Lee K.Abe N.Pu C.Qiao M.Ahmed N.Kossmann D.Saltz J.Tang J.He J.Liu H.Hu X.","Baidu;et al.;Expedia Group;IEEE;IEEE Computer Society;Squirrel AI Learning","Institute of Electrical and Electronics Engineers Inc.","2018 IEEE International Conference on Big Data, Big Data 2018","10 December 2018 through 13 December 2018",,144531,,9781538650356,,,"English","Proc. - IEEE Int. Conf. Big Data, Big Data",Conference Paper,"Final","All Open Access, Green",Scopus,2-s2.0-85062597161
"Liu J., Li W., Pei H., Wang Y., Qu F., Qu Y., Chen Y.","57195587411;56510367700;56719189500;56108225800;57214509597;55933452400;57195586540;","Identity Preserving Generative Adversarial Network for Cross-Domain Person Re-Identification",2019,"IEEE Access","7",,"8792068","114021","114032",,6,"10.1109/ACCESS.2019.2933910","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85082028028&doi=10.1109%2fACCESS.2019.2933910&partnerID=40&md5=2df23b43f0a57685464e782b2c74f84a","College of Computer Science and Technology, Jilin University, Changchun, China","Liu, J., College of Computer Science and Technology, Jilin University, Changchun, China; Li, W., College of Computer Science and Technology, Jilin University, Changchun, China; Pei, H., College of Computer Science and Technology, Jilin University, Changchun, China; Wang, Y., College of Computer Science and Technology, Jilin University, Changchun, China; Qu, F., College of Computer Science and Technology, Jilin University, Changchun, China; Qu, Y., College of Computer Science and Technology, Jilin University, Changchun, China; Chen, Y., College of Computer Science and Technology, Jilin University, Changchun, China","In this paper, we study the domain adaptive person re-identification(re-ID) problem: train a re-ID model on the labeled source domain and test it on the unlabeled target domain. It's known challenging due to the feature distribution bias between the source domain and target domain. The previous methods directly reduce the bias by image-to-image style translation between the source and the target domain in an unsupervised manner. However, these methods only consider the rough bias between the source domain and the target domain but neglect the detailed bias between the source domain and the target camera domains (divided by camera views), which contain critical factors influencing the testing performance of re-ID model. In this work, we particularly focus on the bias between the source domain and the target camera domains. To overcome this problem, a multi-domain image-to-image translation network, termed Identity Preserving Generative Adversarial Network (IPGAN) is proposed to learn the mapping relationship between the source domain and the target camera domains. IPGAN can translate the styles of images from the source domain to the target camera domains and generate many images with styles of target camera domains. Then the re-ID model is trained with the translated images generated by IPGAN. During the training of the re-ID model, we aim to learn the discriminative feature. We design and train a novel re-ID model, termed IBN-reID, in which Instance and Batch Normalization block (IBN-block) are introduced. Experimental results on Market-1501, DukeMTMC-reID and MSMT17 show that the images generated by IPGAN are more suitable for cross-domain re-ID. Very competitive re-ID accuracy is achieved by our method. © 2013 IEEE.","domain adaptation; Person re-identification; style transfer; unsupervised learning","Adversarial networks; Critical factors; Discriminative features; Feature distribution; Image translation; Mapping relationships; Person re identifications; Testing performance; Cameras",,,,,"20170204020GX

Jilin Province Development and Reform Commission: 2019C054-2

National Natural Science Foundation of China, NSFC: 5180523","This work was supported in part by the Science and Technology Development Plan of Jilin Province under Grant 20170204020GX, in part by the Development and Reform Commission of Jilin Province under Grant 2019C054-2, and in part by the National Natural Science Foundation of China under Grant 5180523.",,"Chen, Y., Zhang, F., Zuo, W., Deep image annotation and classification by fusing multi-modal semantic topics (2018) KSII Trans. Internet Inf. Syst., 12 (1), pp. 392-412. , Jan; Luo, F., Du, B., Zhang, L., Zhang, L., Tao, D., Feature learning using spatial-spectral Hypergraph discriminant analysis for hyperspectral image (2019) IEEE Trans. Cybern., 49 (7), pp. 2406-2419. , Jul; Pang, S., Del Coz, J.J., Yu, Z., Luaces, O., Díez, J., Deep learning and preference learning for object tracking: A combined approach (2018) Neural Process. Lett., 47 (3), pp. 859-876. , Jun; Wang, S.-J., Chen, H.-L., Yan, W.-J., Chen, Y.-H., Fu, X., Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine (2014) Neural Process. Lett., 39 (1), pp. 25-43; Zhang, L., Zhang, L., Du, B., You, J., Tao, D., Hyperspectral image unsupervised classification by robust manifold matrix factorization (2019) Inf. Sci., 485, pp. 154-169. , Jun; Zhu, F., Liu, Q., Zhong, S., Yang, Y., Human-level moving object recognition from traffic video (2015) Comput. Sci. Inf. Syst., 12 (2), pp. 787-799; Liao, S., Hu, Y., Zhu, X., Li, S.Z., Person re-identification by local maximal occurrence representation and metric learning (2015) Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 2197-2206. , Jun; Zhang, L., Xiang, T., Gong, S., Learning a discriminative null space for person re-identification (2016) Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 1239-1248. , Jun; Zheng, L., Yang, Y., Hauptmann, A.G., (2016) Person Re-identification: Past, Present and Future, , https://arxiv.org/abs/1610.02984, arXiv:1610. 02984; Sun, Y., Zheng, L., Deng, W., Wang, S., (2017) SVDNet for Pedestrian Retrieval, , https://arxiv.org/abs/1703.05693, arXiv:1703. 05693; Hermans, A., Beyer, L., Leibe, B., (2017) In Defense of the Triplet Loss for Person Re-identification, , https://arxiv.org/abs/1703.07737, arXiv:1703. 07737; Sun, Y., Zheng, L., Yang, Y., Tian, Q., Wang, S., (2017) Beyond Part Models: Person Retrieval with Refined Part Pooling, , https://arxiv.org/abs/1711.09349, arXiv:1711. 09349; Wei, L., Zhang, S., Gao, W., Tian, Q., Person transfer GAN to bridge domain gap for person re-identification (2018) Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 79-88. , Jun; Deng, W., Zheng, L., Kang, G., Yang, Y., Ye, Q., Jiao, J., Image-image domain adaptation with preserved self-similarity and domain-dissimilarity for person re-identification (2018) Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 1 (2), pp. 994-1003. , Jun; Fan, H., Zheng, L., Yan, C., Yang, Y., Unsupervised person reidenti fication: Clustering and fine-tuning (2018) ACM Trans. Multimedia Com-put. Commun. Appl., 14 (4), p. 83. , Oct; Peng, P., Xiang, T., Wang, Y., Pontil, M., Gong, S., Huang, T., Tian, Y., Unsupervised cross-dataset transfer learning for person reidenti fication (2016) Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 1306-1315. , Jun; Yu, H.-X., Wu, A., Zheng, W.-S., Cross-view asymmetric metric learning for unsupervised person re-identification (2017) Proc. IEEE Int. Conf. Comput. Vis. (ICCV), pp. 994-1002. , Oct; Zheng, Z., Zheng, L., Yang, Y., (2017) Unlabeled Samples Generated by Gan Improve the Person Re-identification Baseline in Vitro, , https://arxiv.org/abs/1701.07717, arXiv:1701. 07717; Ristani, E., Solera, F., Zou, R., Cucchiara, R., Tomasi, C., Performance measures and a data set for multi-target, multi-camera tracking (2016) Proc. Eur. Conf. Comput. Vis. Cham, Switzerland: Springer, pp. 17-35; Zheng, L., Shen, L., Tian, L., Wang, S., Wang, J., Tian, Q., Scalable person re-identification: A benchmark (2015) Proc. IEEE Int. Conf. Comput. Vis. (ICCV), pp. 1116-1124. , Dec; Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., Choo, J., (2017) Star-GAN: Unified Generative Adversarial Networks for Multi-domain Imageto-image Translation, , https://arxiv.org/abs/1711.09020, arXiv:1711. 09020; Pan, X., Luo, P., Shi, J., Tang, X., (2018) Two at Once: Enhancing Learning and Generalization Capacities Via IBN-net, , https://arxiv.org/abs/1807.09441, arXiv:1807. 09441; Liu, M.-Y., Tuzel, O., Coupled generative adversarial networks (2016) Proc. Adv. Neural Inf. Process. Syst, pp. 469-477; Wang, J., Zhu, X., Gong, S., Li, W., Transferable joint attribute-identity deep learning for unsupervised person re-identification (2018) Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 2275-2284. , Jun; Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A., (2017) Image-to-image Translation with Conditional Adversarial Networks, , https://arxiv.org/abs/1611.07004, arXiv:1611. 07004; Zhu, J.-Y., Park, T., Isola, P., Efros, A.A., (2017) Unpaired Image-toimage Translation Using Cycle-consistent Adversarial Networks, , https://arxiv.org/abs/1703.10593, arXiv:1703. 10593; Kim, T., Cha, M., Kim, H., Lee, J.K., Kim, J., (2017) Learning to Discover Cross-domain Relations with Generative Adversarial Networks, , https://arxiv.org/abs/1703.05192, arXiv:1703. 05192; Liu, M.-Y., Breuel, T., Kautz, J., Unsupervised image-to-image translation networks (2017) Proc. Adv. Neural Inf. Process. Syst., pp. 700-708; Kingma, D.P., Welling, M., (2013) Auto-encoding Variational Bayes, , https://arxiv.org/abs/1312.6114, arXiv:1312. 6114; He, K., Zhang, X., Ren, S., Sun, J., Deep residual learning for image recognition (2016) Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 770-778. , Jun; Wang, F., Zuo, W., Lin, L., Zhang, D., Zhang, L., Joint learning of singleimage and cross-image representations for person re-identification (2016) Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 1288-1296. , Jun; Xiao, T., Li, H., Ouyang, W., Wang, X., Learning deep feature representations with domain guided dropout for person re-identification (2016) Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 1249-1258; Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Proc. Int. Conf. Neural Inf. Process. Syst., pp. 2672-2680; Liu, Z., Wang, D., Lu, H., Stepwise metric promotion for unsupervised video person re-identification (2017) Proc. IEEE Int. Conf. Comput. Vis. (ICCV), pp. 2448-2457. , Oct; Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D., Object detection with discriminatively trained part-based models (2010) IEEE Trans. Pattern Anal. Mach. Intell., 32 (9), pp. 1627-1645. , Sep; Zhong, Z., Zheng, L., Zhenga, Z., Li, S., Yang, Y., CamStyle: A novel data augmentation method for person re-identification (2018) IEEE Trans. Image Process., 28 (3), pp. 1176-1190. , Mar; Subramaniam, A., Chatterjee, M., Mittal, A., Deep neural networks with inexact matching for person re-identification (2016) Proc. Adv. Neural Inf. Process. Syst., pp. 2667-2675; Song, J., Yang, Y., Song, Y.-Z., Xiang, T., Hospedales, T.M., Generalizable person re-identification by domain-invariant mapping network (2019) Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 719-728. , Jun; Song, L., Wang, C., Zhang, L., Du, B., Zhang, Q., Huang, C., Wang, X., Unsupervised domain adaptive re-identification: Theory and practice 2018, arXiv:1807. 11334, , https://arxiv.org/abs/1807.11334; Zhong, Z., Zheng, L., Luo, Z., Li, S., Yang, Y., Invariance matters: Exemplar memory for domain adaptive person re-identification (2019) Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 598-607. , Jun","Li, W.; College of Computer Science and Technology, China; email: liwh@jlu.edu.cn",,,"Institute of Electrical and Electronics Engineers Inc.",,,,,21693536,,,,"English","IEEE Access",Article,"Final","All Open Access, Gold, Green",Scopus,2-s2.0-85082028028
"Xu D., Wu Y., Yuan S., Zhang L., Wu X.","57201947036;57191480281;56486630300;57191309823;8426129400;","Achieving causal fairness through generative adversarial networks",2019,"IJCAI International Joint Conference on Artificial Intelligence","2019-August",,,"1452","1458",,4,"10.24963/ijcai.2019/201","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85074944101&doi=10.24963%2fijcai.2019%2f201&partnerID=40&md5=d15ce9cd27864e470b1fff01420a136d","University of Arkansas, United States","Xu, D., University of Arkansas, United States; Wu, Y., University of Arkansas, United States; Yuan, S., University of Arkansas, United States; Zhang, L., University of Arkansas, United States; Wu, X., University of Arkansas, United States","Achieving fairness in learning models is currently an imperative task in machine learning. Meanwhile, recent research showed that fairness should be studied from the causal perspective, and proposed a number of fairness criteria based on Pearl's causal modeling framework. In this paper, we investigate the problem of building causal fairness-aware generative adversarial networks (CFGAN), which can learn a close distribution from a given dataset, while also ensuring various causal fairness criteria based on a given causal graph. CFGAN adopts two generators, whose structures are purposefully designed to reflect the structures of causal graph and interventional graph. Therefore, the two generators can respectively simulate the underlying causal model that generates the real data, as well as the causal model after the intervention. On the other hand, two discriminators are used for producing a close-to-real distribution, as well as for achieving various fairness criteria based on causal quantities simulated by generators. Experiments on a real-world dataset show that CFGAN can generate high quality fair data. © 2019 International Joint Conferences on Artificial Intelligence. All rights reserved.",,,,,,,"National Science Foundation, NSF: 1564250, 1646654, 1841119","This work was supported in part by NSF 1646654, 1564250, and 1841119.",,"Calders, T., Kamiran, F., Pechenizkiy, M., Building classifiers with independency constraints (2009) 2009 IEEE International Conference on Data Mining Workshops; Chiappa, S., Path-specific counterfactual fairness (2019) The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19); Dheeru, D., Taniskidou, E.K., (2017) UCI Machine Learning Repository, , University of California, Irvine, School of Information and Computer Sciences; Edwards, H., Storkey, A.J., Censoring representations with an adversary (2016) 4th International Conference on Learning Representations, ICLR 2016, , San Juan, Puerto Rico; Feldman, M., Friedler, S.A., Moeller, J., Scheidegger, C., Venkatasubramanian, S., Certifying and removing disparate impact (2015) Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD'15, , New York, NY, USA; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., Generative adversarial nets (2014) Advances in Neural Information Processing Systems 27 (NIPS 2014; Hardt, M., Price, E., None, Srebro, N., Equality of opportunity in supervised learning (2016) Advances in Neural Information Processing Systems 29 (NIPS, , 2016; Kamiran, F., Calders, T., Classifying without discriminating (2009) 2009 2nd International Conference on Computer, Control and Communication, pp. 1-6. , IEEE, February; Kamiran, F., Calders, T., Data preprocessing techniques for classification without discrimination (2012) Knowledge and Information Systems, 33 (1), pp. 1-33. , October; Kocaoglu, M., Snyder, C., Dimakis, A.G., Vishwanath, S., Causalgan: Learning causal implicit generative models with adversarial training (2018) 6th International Conference on Learning Representations; Kusner, M.J., Loftus, J., Russell, C., Silva, R., Counterfactual fairness (2017) Advances in Neural Information Processing Systems; Madras, D., Creager, E., Pitassi, T., Zemel, R.S., Learning adversarially fair and transferable representations (2018) Proceedings of the 35th International Conference on Machine Learning, , Stockholmsmässan, Stockholm, Sweden; Nabi, R., Shpitser, I., Fair inference on outcomes (2018) The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18); Pearl, J., (2009) Causality: Models, Reasoning and Inference, , Cambridge University Press, New York, NY, USA, 2nd edition; Pedreshi, D., Ruggieri, S., Turini, F., Discrimination-aware data mining (2008) Proceeding of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD 08, , New York, New York, USA; Salimi, B., Rodriguez, L., Howe, B., Suciu, D., (2019) Capuchin: Causal Database Repair for Algorithmic Fairness, , CoRR, abs/1902.08283; Wu, Y., Zhang, L., Wu, X., Counterfactual fairness: Unidentification, bound and algorithm (2019) Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, , Macao, China, August 10-16, 2019; Xie, Q., Dai, Z., Du, Y., Hovy, E., Neubig, G., Controllable invariance through adversarial feature learning (2017) Advances in Neural Information Processing Systems 30 NIPS, , 2017; Xu, D., Yuan, S., Zhang, L., Wu, X., Fairgan: Fairness-aware generative adversarial networks (2018) IEEE International Conference on Big Data, Big Data 2018, pp. 570-575. , Seattle, WA, USA, December 10-13, 2018; Zhang, J., Bareinboim, E., Fairness in decision-making - The causal explanation formula (2018) The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18); Zhang, L., Wu, Y., Wu, X., A causal framework for discovering and removing direct and indirect discrimination (2017) Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, , Melbourne, Australia; Zhang, B.H., Lemoine, B., Mitchell, M., Mitigating unwanted biases with adversarial learning (2018) Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES 2018, , New Orleans, LA, USA; Zhang, L., Wu, Y., Wu, X., Achieving non-discrimination in prediction (2018) Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, pp. 3097-3103. , July 13-19, 2018, Stockholm, Sweden; Zhang, L., Wu, Y., Wu, X., Causal modeling-based discrimination discovery and removal: Criteria, bounds, and algorithms (2018) IEEE Transactions on Knowledge and Data Engineering; Zliobaite, I., Kamiran, F., Calders, T., Handling conditional discrimination (2011) 2011 IEEE 11th International Conference on Data Mining",,"Kraus S.","Baidu;et al.;Huawei;International Joint Conferences on Artifical Intelligence (IJCAI);Sony;Xiao-i","International Joint Conferences on Artificial Intelligence","28th International Joint Conference on Artificial Intelligence, IJCAI 2019","10 August 2019 through 16 August 2019",,153611,10450823,9780999241141,,,"English","IJCAI Int. Joint Conf. Artif. Intell.",Conference Paper,"Final","All Open Access, Bronze",Scopus,2-s2.0-85074944101
"Gadermayr M., Klinkhammer B.M., Boor P.","50361333000;55596800700;16314805700;","Virtually Redying Histological Images with Generative Adversarial Networks to Facilitate Unsupervised Segmentation: A Proof-of-Concept Study",2019,"Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)","11435 LNCS",,,"38","46",,,"10.1007/978-3-030-23937-4_5","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85069199967&doi=10.1007%2f978-3-030-23937-4_5&partnerID=40&md5=990fe79e056c0da8ef844f290c4fc60c","Salzburg University of Applied Sciences, Salzburg, Austria; Institute of Imaging and Computer Vision, RWTH Aachen University, Aachen, Germany; Institute Pathology, University Hospital Aachen, RWTH Aachen University, Aachen, Germany","Gadermayr, M., Salzburg University of Applied Sciences, Salzburg, Austria, Institute of Imaging and Computer Vision, RWTH Aachen University, Aachen, Germany; Klinkhammer, B.M., Institute Pathology, University Hospital Aachen, RWTH Aachen University, Aachen, Germany; Boor, P., Institute Pathology, University Hospital Aachen, RWTH Aachen University, Aachen, Germany","Approaches relying on adversarial networks facilitate image-to-image-translation based on unpaired training and thereby open new possibilities for special tasks in image analysis. We propose a methodology to improve segmentability of histological images by making use of image-to-image translation. We generate virtual stains and exploit the additional information during segmentation. Specifically a very basic pixel-based segmentation approach is applied in order to focus on the information content available on pixel-level and to avoid any bias which might be introduced by more elaborated techniques. The results of this proof-of-concept trial indicate a performance gain compared to segmentation with the source stain only. Further experiments including more powerful supervised state-of-the-art machine learning approaches and larger evaluation data sets need to follow. © 2019, Springer Nature Switzerland AG.","Adversarial networks; Glomeruli; Histology; Kidney; Segmentation; Tubuli; Unsupervised","Histology; Image enhancement; Pathology; Pixels; Adversarial networks; Glomeruli; Kidney; Tubuli; Unsupervised; Image segmentation",,,,,,,,"Barker, J., Hoogi, A., Depeursinge, A., Rubin, D.L., Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles (2016) Med. Image Anal., 30, pp. 60-71; Bentaieb, A., Hamarneh, G., Topology aware fully convolutional networks for histology gland segmentation (2016) MICCAI 2016. LNCS, 9901, pp. 460-468. , https://doi.org/10.1007/978-3-319-46723-853, Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.), pp., Springer, Cham; Gadermayr, M., Appel, V., Klinkhammer, B.M., Boor, P., Merhof, D., Which way round? A study on the performance of stain-translation for segmenting arbitrarily dyed histological images (2018) MICCAI 2018. LNCS, 11071, pp. 165-173. , https://doi.org/10.1007/978-3-030-00934-219, Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.), pp., Springer, Cham; Gadermayr, M., Dombrowski, A.K., Klinkhammer, B.M., Boor, P., Merhof, D., CNN cascades for segmenting sparse objects in gigapixel whole slide images (2019) Comput. Med. Imaging Graph., 71, pp. 40-48; Gadermayr, M., Eschweiler, D., Jeevanesan, A., Klinkhammer, B.M., Boor, P., Merhof, D., Segmenting renal whole slide images virtually without training data (2017) Comput. Biol. Med., 90, pp. 88-97; Hou, L., Samaras, D., Kurc, T.M., Gao, Y., Davis, J.E., Saltz, J.H., Patch-based convolutional neural network for whole slide tissue image classification (2016) Proceedings of the International Conference on Computer Vision (CVPR 2016; Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A., Image-to-image translation with conditional adversarial networks (2017) In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR; Johnson, J., Alahi, A., Fei-Fei, L., Perceptual losses for real-time style transfer and super-resolution (2016) ECCV 2016. LNCS, 9906, pp. 694-711. , https://doi.org/10.1007/978-3-319-46475-643, In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.); Macenko, M., A method for normalizing histology slides for quantitative analysis (2009) Proceedings of the IEEE International Symposium on Biomedical Imaging: From Nano to Macro, pp. 1107-1110. , https://doi.org/10.1109/ISBI.2009.5193250, ISBI 2009), pp; Ronneberger, O., Fischer, P., Brox, T., U-Net: Convolutional networks for biomedical image segmentation (2015) MICCAI 2015. LNCS, 9351, pp. 234-241. , https://doi.org/10.1007/978-3-319-24574-428, Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.), pp., Springer, Cham; Sertel, O., Kong, J., Shimada, H., Catalyurek, U.V., Saltz, J.H., Gurcan, M.N., Computer-aided prognosis of neuroblastoma on whole-slide images: Classification of stromal development (2009) Pattern Recognit, 42 (6), pp. 1093-1103; Zhu, J.Y., Park, T., Isola, P., Efros, A.A., Unpaired image-to-image translation using cycle-consistent adversarial networks (2017) In: Proceedings of the International Conference on Computer Vision (ICCV 2017)","Gadermayr, M.; Salzburg University of Applied SciencesAustria; email: Michael.Gadermayr@fh-salzburg.ac.at","Reyes-Aldasoro C.C.Janowczyk A.Veta M.Bankhead P.Sirinukunwattana K.",,"Springer Verlag","15th European Congress on Digital Pathology, ECDP 2019","10 April 2019 through 13 April 2019",,228379,03029743,9783030239367,,,"English","Lect. Notes Comput. Sci.",Conference Paper,"Final","",Scopus,2-s2.0-85069199967
"Xu H., Cao Y., Jia R., Liu Y., Tan J.","57202685620;55431244700;57205618552;36976018300;13606951900;","Sequence generative adversarial network for long text summarization",2018,"Proceedings - International Conference on Tools with Artificial Intelligence, ICTAI","2018-November",,"8576043","242","248",,1,"10.1109/ICTAI.2018.00045","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85060816978&doi=10.1109%2fICTAI.2018.00045&partnerID=40&md5=5169bfb1e5c99a10505cd2441f253e8f","School of Cyber Security, University of Chinese Academy of Sciences, Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China; Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China","Xu, H., School of Cyber Security, University of Chinese Academy of Sciences, Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China; Cao, Y., Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China; Jia, R., School of Cyber Security, University of Chinese Academy of Sciences, Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China; Liu, Y., Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China; Tan, J., Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China","In this paper, we propose a new adversarial training framework for text summarization task. Although sequence-to-sequence models have achieved state-of-the-art performance in abstractive summarization, the training strategy (MLE) suffers from exposure bias in the inference stage. This discrepancy between training and inference makes generated summaries less coherent and accuracy, which is more prominent in summarizing long articles. To address this issue, we model abstractive summarization using Generative Adversarial Network (GAN), aiming to minimize the gap between generated summaries and the ground-truth ones. This framework consists of two models: a generator that generates summaries, a discriminator that evaluates generated summaries. Reinforcement learning (RL) strategy is used to guarantee the co-training of generator and discriminator. Besides, motivated by the nature of summarization task, we design a novel Triple-RNNs discriminator, and extend the off-the-shelf generator by appending encoder and decoder with attention mechanism. Experimental results showed that our model significantly outperforms the state-of-the-art models, especially on long text corpus. © 2018 IEEE.","Deep learning; Reinforcement learning; Sequence Generative Adversarial Network; Text Summarization","Deep learning; Machine learning; Reinforcement learning; Adversarial networks; Attention mechanisms; Inference stages; State of the art; State-of-the-art performance; Text summarization; Training framework; Training strategy; Text processing",,,,,,,,"Nallapati, R., Zhou, B., Dos Santos, C., Gulçehre Ç.glar, Xiang, B., Abstractive text summarization using sequence-to-sequence rnns and beyond (2016) CoNLL 2016, p. 280; Chopra, S., Auli, M., Rush, A.M., Abstractive sentence summarization with attentive recurrent neural networks (2016) Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 93-98; Bengio, S., Vinyals, O., Jaitly, N., Scheduled sampling for sequence prediction with recurrent neural networks (2015) Advances in Neural Information Processing Systems, pp. 1171-1179; Wang, S., Zhao, X., Li, B., Integrating extractive and abstractive models for long text summarization (2017) Big Data (BigData Congress), 2017 IEEE International Congress On. IEEE, pp. 305-312; Liu, S., (2017) Cs585 Project Report Long Text Summarization Using Neural Networks and Rule-based Approach; Wu, L., Xia, Y., Zhao, L., (2017) Adversarial Neural Machine Translation, , arXiv preprint arXiv: 1704.06933; Yu, L., Zhang, W., Wang, J., Yu, Y., Seqgan: Sequence generative adversarial nets with policy gradient (2017) AAAI, pp. 2852-2858; Conroy, J.M., O'Leary, D.P., Text summarization via hidden markov models (2001) Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, pp. 406-407; Ferrier, L., (2001) A Maximum Entropy Approach to Text Summarization, , School of Artificial Intelligence, Division of Informatics, University of Edinburgh; Mihalcea, R., Tarau, P., Textrank: Bringing order into text (2004) Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing; Erkan, G., Radev, D.R., Lexrank: Graph-based lexical centrality as salience in text summarization (2004) Journal of Artificial Intelligence Research, 22, pp. 457-479; Zajic, D., Dorr, B., Schwartz, R., Bbn/umd at duc-2004: Topiary (2004) Proceedings of the HLT-NAACL, 2004 Document Understanding Workshop, pp. 112-119. , Boston; Cohn, T., Lapata, M., Sentence compression beyond word deletion (2008) Proceedings of the 22nd International Conference on Computational Linguistics, 1, pp. 137-144. , Association for Computational Linguistics; Kalchbrenner, N., Blunsom, P., Recurrent continuous translation models (2013) Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1700-1709; Sutskever, I., Vinyals, O., Le, Q.V., Sequence to sequence learning with neural networks (2014) Advances in Neural Information Processing Systems, pp. 3104-3112; Rush, A.M., Chopra, S., Weston, J., (2015) A Neural Attention Model for Abstractive Sentence Summarization, pp. 379-389; Nallapati, R., Xiang, B., Zhou, B., (2016) Sequence-to-sequence Rnns for Text Summarization; Liu, P.J., Pan, X., Text summarization with tensorflow (2016) Google Research Blog. Google Brain Team, 24; Goodfellow, I., Pouget-Abadie, J., Mirza, M., Generative adversarial nets (2014) Advances in Neural Information Processing Systems, pp. 2672-2680; Huszár, F., How (not) to train your generative model: Scheduled sampling, likelihood, adversary? (2015) Stat, 1050, p. 16; Denton, E.L., Chintala, S., Fergus, R., Deep generative image models using a laplacian pyramid of adversarial networks (2015) Advances in Neural Information Processing Systems, pp. 1486-1494; Liu, L., Lu, Y., Yang, M., (2017) Generative Adversarial Network for Abstractive Text Summarization, , arXiv preprint arXiv: 1711.09357; Wang, B., Liu, K., Zhao, J., Inner attention based recurrent neural networks for answer selection (2016) Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1, pp. 1288-1297; Wu, Y., Bamman, D., Russell, S., Adversarial training for relation extraction (2017) Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1778-1783; Yin, W., Kann, K., Yu, M., Schütze, H., Comparative Study of Cnn and Rnn for Natural Language Processing, p. 2017. , arXiv preprint arXiv: 1702.01923; Hermann, K.M., Kocisky, T., Grefenstette, E., Teaching machines to read and comprehend (2015) Advances in Neural Information Processing Systems, pp. 1693-1701; Lin, C.-Y., Rouge: A package for automatic evaluation of summaries (2004) Text Summarization Branches Out; Paulus, R., Xiong, C., Socher, R., (2017) A Deep Reinforced Model for Abstractive Summarization, , arXiv preprint arXiv: 1705.04304; Kingma, D.P., Ba, J., (2014) Adam: A Method for Stochastic Optimization, , arXiv preprint arXiv: 1412.6980; Abadi, M., Agarwal, A., Barham, P., (2016) Tensorflow: Large-scale Machine Learning on Heterogeneous Distributed Systems, , arXiv preprint arXiv: 1603.04467",,,"Biological and Artificial Intelligence Foundation (BAIF);IEEE Computer Society","IEEE Computer Society","30th International Conference on Tools with Artificial Intelligence, ICTAI 2018","5 November 2018 through 7 November 2018",,143794,10823409,9781538674499,PCTIF,,"English","Proc. Int. Conf. Tools Artif. Intell. ICTAI",Conference Paper,"Final","",Scopus,2-s2.0-85060816978
"Zhang Z., Liu S., Li M., Zhou M., Chen E.","57202958159;56181265100;35110581300;55587890800;35228685900;","Bidirectional generative adversarial networks for neural machine translation",2018,"CoNLL 2018 - 22nd Conference on Computational Natural Language Learning, Proceedings",,,,"190","199",,5,"10.18653/v1/k18-1019","https://www.scopus.com/inward/record.uri?eid=2-s2.0-85072935809&doi=10.18653%2fv1%2fk18-1019&partnerID=40&md5=121298a91744bbde719be83b200b1b13","University of Science and Technology of China, Hefei, China; Microsoft Research Asia, China","Zhang, Z., University of Science and Technology of China, Hefei, China; Liu, S., Microsoft Research Asia, China; Li, M.; Zhou, M., Microsoft Research Asia, China; Chen, E., University of Science and Technology of China, Hefei, China","Generative Adversarial Network (GAN) has been proposed to tackle the exposure bias problem of Neural Machine Translation (NMT). However, the discriminator typically results in the instability of the GAN training due to the inadequate training problem: the search space is so huge that sampled translations are not sufficient for discriminator training. To address this issue and stabilize the GAN training, in this paper, we propose a novel Bidirectional Generative Adversarial Network for Neural Machine Translation (BGAN-NMT), which aims to introduce a generator model to act as the discriminator, whereby the discriminator naturally considers the entire translation space so that the inadequate training problem can be alleviated. To satisfy this property, generator and discriminator are both designed to model the joint probability of sentence pairs, with the difference that, the generator decomposes the joint probability with a source language model and a source-to-target translation model, while the discriminator is formulated as a target language model and a target-to-source translation model. To further leverage the symmetry of them, an auxiliary GAN is introduced and adopts generator and discriminator models of original one as its own discriminator and generator respectively. Two GANs are alternately trained to update the parameters. Experiment results on German-English and Chinese-English translation tasks demonstrate that our method not only stabilizes GAN training but also achieves significant improvements over baseline systems. © 2018 Association for Computational Linguistics.",,"Computer aided language translation; Natural language processing systems; Adversarial networks; Baseline systems; Generator modeling; Joint probability; Machine translations; Source language; Target language; Translation models; Computational linguistics",,,,,,,,"Arjovsky, M., Chintala, S., Bottou, L., Wasserstein gagn (2017) CoRR; Artetxe, M., Labaka, G., Agirre, E., Cho, K., Unsupervised neural machine translation (2017) CoRR; Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R., Pineau, J., Courville, A.C., Bengio, Y., An actor-critic algorithm for sequence prediction (2016) CoRR; Bahdanau, D., Cho, K., Bengio, Y., Neural machine translation by jointly learning to align and translate (2014) CoRR; Bengio, S., Vinyals, O., Jaitly, N., Shazeer, N., Scheduled sampling for sequence prediction with recurrent neural networks (2015) NIPS; Cheng, Y., Xu, W., He, Z., He, W., Wu, H., Sun, M., Liu, Y., Semi-supervised learning for neural machine translation (2016) ACL; Chiang, D., Hierarchical phrase-based translation (2007) Computational Linguistics, 33, pp. 201-228; Cho, K., van Merrienboer, B., Gulçehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y., Learning phrase representations using rnn encoder-decoder for statistical machine translation (2014) EMNLP; Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y., Convolutional sequence to sequence learning (2017) ICML; Glorot, X., Bengio, Y., Understanding the difficulty of training deep feedforward neural networks (2010) AISTATS; Goodfellow, I.J., Nips 2016 tutorial: Generative adversarial networks (2016) CoRR; Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A.C., Bengio, Y., Generative adversarial nets (2014) NIPS; Hassan, H., Aue, A., Chen, C., Chowdhary, V., Clark, J., Federmann, C., Huang, X., Zhou, M., Achieving human parity on automatic chinese to english news translation (2018) CoRR; He, D., Xia, Y., Qin, T., Wang, L., Yu, N., Liu, T.-Y., Ma, W.-Y., Dual learning for machine translation (2016) NIPS; Hochreiter, S., Schmidhuber, J., Long short-term memory (1997) Neural Computation; Koehn, P., Och, F.J., Marcu, D., Statistical phrase-based translation (2003) HLTNAACL; Lamb, A., Goyal, A., Zhang, Y., Zhang, S., Courville, A.C., Bengio, Y., Professor forcing: A new algorithm for training recurrent networks (2016) NIPS; Lample, G., Ott, M., Conneau, A., Denoyer, L., Ranzato, M., Phrase-based & neural unsupervised machine translation (2018) CoRR; Li, J., Monroe, W., Shi, T., Ritter, A., Jurafsky, D., Adversarial learning for neural dialogue generation (2017) EMNLP; Luong, T., Sutskever, I., Le, Q.V., Vinyals, O., Zaremba, W., Addressing the rare word problem in neural machine translation (2015) ACL; Ma, X., Yin, P., Liu, J., Neubig, G., Hovy, E.H., Soft-max q-distribution estimation for structured prediction: A theoretical interpretation for raml (2017) CoRR; Papineni, K., Roucos, S.E., Ward, T., Zhu, W.-J., BLEU: A method for automatic evaluation of machine translation (2002) ACL; Radford, A., Metz, L., Chintala, S., (2015) Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, , arXiv preprint; Ranzato, M., Chopra, S., Auli, M., Zaremba, W., Sequence level training with recurrent neural networks (2015) CoRR; Sennrich, R., Haddow, B., Birch, A., Improving neural machine translation models with monolingual data (2016) ACL; Shen, S., Cheng, Y., He, Z., He, W., Wu, H., Sun, M., Liu, Y., Minimum risk training for neural machine translation (2016) ACL; Sutskever, I., Vinyals, O., Le, Q.V., Sequence to sequence learning with neural networks (2014) NIPS; Tu, Z., Liu, Y., Shang, L., Liu, X., Li, H., Neural machine translation with reconstruction (2017) AAAI; Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I., Attention is all you need (2017) NIPS; Wiseman, S., Rush, A.M., Sequence-to-sequence learning as beam-search optimization (2016) EMNLP; Wu, L., Xia, Y., Zhao, L., Tian, F., Qin, T., Lai, J.-H., Liu, T.-Y., Adversarial neural machine translation (2017) CoRR; Yang, Z., Chen, W., Wang, F., Xu, B., Improving neural machine translation with conditional sequence generative adversarial nets (2017) CoRR; Yu, L., Zhang, W., Wang, J., Yu, Y., SeqGan: Sequence generative adversarial nets with policy gradient (2017) AAAI",,,"Google Inc.;Textkernel","Association for Computational Linguistics (ACL)","22nd Conference on Computational Natural Language Learning, CoNLL 2018","31 October 2018 through 1 November 2018",,150065,,9781948087728,,,"English",,Conference Paper,"Final","All Open Access, Hybrid Gold",Scopus,2-s2.0-85072935809