Extended UCB Policies for Multi-armed Bandit Problems
Published by
Machine Learning
Summary
Regret minimization by upper confidence bound for general reward distributions.
Keqin (Kevin) Liu
Associate Professor & AI/Systems Architect
Suzhou, CN.About
Highly accomplished Associate Professor and AI/Systems Architect with over 15 years of interdisciplinary expertise spanning advanced mathematics, artificial intelligence, and complex systems engineering. Recognized for pioneering research in multi-armed bandits and stochastic optimization, evidenced by 40+ top-tier publications and 2700+ citations, alongside significant contributions to chip-manufacturing and autonomous systems as a Senior/Staff Software Engineer and CTO. Proven leader in academia, developing and delivering high-impact curricula to 800+ students with top evaluation scores, and driving innovative R&D initiatives in AI e-commerce.
Work
XJTLU
|Associate Professor, School of Math. & Phys.
Suzhou, Jiangsu, China
→
Summary
Leading advanced research and instruction in pure and applied mathematics, contributing to academic innovation and student development.
Highlights
Initiated and will lead advanced research projects in pure and applied mathematics, fostering interdisciplinary innovation and contributing to the academic community.
Developed and will deliver engaging curricula for undergraduate and graduate students, leveraging deep expertise in mathematical theory and AI applications.
Mentored and supervised student research, guiding the next generation of mathematicians and scientists in cutting-edge fields.
JS National Center for Applied Math.
|Research Fellow
China
→
Summary
Conducted cutting-edge research on number theory and AI-related topics, contributing to foundational advancements in applied mathematics with significant publications.
Highlights
Conducted advanced research in number theory and AI, resulting in 40+ top international journal and conference papers and over 2700 citations.
Secured significant research funding, including the SIP International Leading Talent (¥400,000 fund) and Zheng-Gang Scholar (¥300,000 fund).
Collaborated with interdisciplinary teams to explore novel applications of mathematical theories in artificial intelligence and complex systems.
Nanjing University (NJU)
|Lecturer, Math. Dept. & Nanjing-Helsinki Institute
Nanjing, Jiangsu, China
→
Summary
Instructed and conducted research in pure and applied mathematics, shaping academic discourse and student learning at a leading institution.
Highlights
Taught core courses (Linear Algebra, Calculus, Probability, Statistics) to 800+ undergraduate students, consistently achieving top evaluation scores from 2020-present.
Developed and launched three new, highly-rated courses: "Introduction to AI for Mathematical Sciences" (2026), "Machine Learning: Mathematical Theory and Applications" (2022), and "Stochastic Optimization" (2021).
Received multiple teaching excellence awards, including the Zhi-Jian Award for Excellence in Teaching (2024) and excellence recognition from Nanjing University for course development (2021, 2022).
Selected as instructor for the general course "The Light of Science" (2023), demonstrating pedagogical leadership and significant impact on student engagement.
ASML
|Senior Software Engineer
US
→
Summary
Designed and implemented critical software and algorithms for advanced chip-manufacturing systems, optimizing performance and precision.
Highlights
Developed sophisticated software and algorithms for high-precision chip-manufacturing systems, enhancing operational efficiency and product quality.
Collaborated with cross-functional engineering teams to integrate complex mathematical algorithms into production-grade software solutions.
Contributed to the advancement of semiconductor manufacturing processes through innovative algorithmic design and system optimization.
KLA
|Staff Software Engineer
US
→
Summary
Engineered software and algorithms for complex autonomous systems, driving innovation in advanced technology solutions.
Highlights
Designed and implemented robust software and algorithms for complex autonomous systems, improving system reliability and performance by optimizing core functionalities.
Optimized existing algorithmic frameworks to enhance the efficiency and accuracy of critical system functions, reducing processing time.
Contributed significantly to the full development lifecycle from conceptualization to deployment, ensuring high-quality software delivery for advanced systems.
Aquarian Age Tech
|CTO
→
Summary
Led a high-performing R&D team in developing a new-generation AI E-Commerce Network, driving technological vision and product innovation.
Highlights
Spearheaded the R&D team to successfully develop a new-generation AI E-Commerce Network, establishing a strategic technological roadmap and vision.
Directed the architectural design and implementation of core AI algorithms, significantly enhancing platform capabilities and user experience.
Managed technical operations and strategic planning, aligning product development with market needs and business objectives to drive innovation.
MOOG CROSSBOW
|Staff Software Engineer
US
→
Summary
Developed advanced software and algorithms for high-accuracy navigation systems, ensuring precision and reliability in critical applications.
Highlights
Designed and implemented software and algorithms for high-accuracy navigation systems, improving precision and system robustness in critical applications.
Contributed to the development of real-time data processing and sensor fusion algorithms, critical for enhancing navigation performance.
Collaborated on system architecture and integration, ensuring seamless functionality and reliability of complex navigation solutions.
UC-Davis
|Postdoctoral Researcher, ECE Dept.
Davis, CA, US
→
Summary
Conducted advanced research in statistical optimization, reinforcement learning, and operations research, leading to significant academic publications and awards.
Highlights
Performed statistical optimization, reinforcement learning, and operations research, leading to multiple publications in top-tier journals and conferences.
Authored the Ph.D. dissertation "On Multi-Armed Bandit in Dynamic Systems," which received the Best Doctoral Dissertation Award in Engineering (2012).
Provided the first mathematical completion of Bayesian and frequentist MAB theory, establishing a foundational framework for future research.
Collaborated on research projects that extended multi-armed bandit theory to various applications, including dynamic spectrum access and distributed learning.
UC-Davis
|Lecturer, “Signals and Systems”, ECE Dept.
Davis, CA, US
→
Summary
Developed and delivered comprehensive undergraduate coursework on "Signals and Systems," fostering deep student understanding.
Highlights
Developed and taught the "Signals and Systems" undergraduate course, effectively conveying complex engineering concepts to students.
Designed comprehensive course materials, lectures, and assignments that facilitated student learning and engagement.
UC-Davis
|Research Assistant, ECE Dept.
Davis, CA, US
→
Summary
Supported advanced research in statistical optimization, reinforcement learning, and operations research, contributing to significant academic breakthroughs and publications.
Highlights
Conducted research in statistical optimization, reinforcement learning, and operations research, contributing to foundational work in multi-armed bandit problems.
Collaborated on projects that resulted in key publications, including "Indexability of Restless Bandit Problems and Optimality of Whittle Index for Dynamic Multichannel Access" (2010).
Developed mathematical models and algorithmic frameworks for complex online decision problems, reducing computational complexity for autonomous systems.
UC-Davis
|Teaching Assistant, ECE Dept.
Davis, CA, US
→
Summary
Provided essential academic support to undergraduate students, facilitating learning and problem-solving in engineering courses.
Highlights
Provided comprehensive homework grading and Q&A support for 200+ American undergraduate students, enhancing their understanding of course material.
Assisted professors in course delivery and student engagement, contributing to a positive learning environment.
Received high evaluation scores from students for effective teaching assistance and support.
Education
UC-Davis
Davis, CA, United States of America
→
Ph.D.
Electrical and Computer Engineering
UC-Davis
Davis, CA, United States of America
→
M.S.
Electrical and Computer Engineering
Southeast University
China
→
B.S.
Automation
Awards
Excellent Research-led Learning Supervisor
Awarded By
XJTLU, China
Anticipated recognition for excellence in research-led learning supervision.
First Prize for Excellent Paper Award
Awarded By
JAAS, China
Awarded first prize for an outstanding paper contribution.
Zhi-Jian Award for Excellence in Teaching
Awarded By
Nanjing University
Received an award for demonstrating excellence in teaching practices.
Conference Chair
Awarded By
CSAMSC, Nanjing, Jiangsu, China
Chaired the CSAMSC conference, overseeing its successful organization and execution.
SIP International Leading Talent
Awarded By
Suzhou, China
Recognized as a leading talent, receiving ¥400,000 in funding for research initiatives.
Zheng-Gang Scholar
Awarded By
Nanjing University
Appointed as a Zheng-Gang Scholar, receiving ¥300,000 in funding (2020-2023).
Zi-Jin Scholar
Awarded By
Nanjing University
Appointed as a Zi-Jin Scholar, receiving ¥800,000 in funding (2020-2023).
Best Doctoral Dissertation Award (Engineering)
Awarded By
UC-Davis
Received for outstanding contributions in the Ph.D. dissertation titled "On Multi-Armed Bandit in Dynamic Systems".
Conference Session Chair
Awarded By
Asilomar, Pacific Grove, CA, USA
Chaired a conference session at Asilomar, demonstrating leadership in academic discourse.
Publications
Generalization and Another Proof of Two Conjectural Identities Posed by Sun
Published by
Linear and Multilinear Algebra
Summary
Number-theoretic approach for solving some trigonometric identities.
Relaxed Indexability and Index Policy for Partially Observable Restless Bandits
Published by
Management Science
Summary
Extension of J8 to arbitrary dimension of belief state space.
Low-complexity Algorithm for Restless Bandits with Imperfect Observations
Published by
Mathematical Methods of Operations Research
Summary
Extension of J8 and J10 to design algorithms for the imperfect observation model.
Approximated Whittle Index for Femtocell Scheduling
Published by
Proc. of The International Conference on Statistics, Applied Mathematics and Computing Science
Summary
Supervised a M.S. student to apply the MAB theory to optimize wireless networks.
Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training
Published by
NeurIPS
Summary
Optimization of layer-wise learning rates in DNN based on stochastic matrix theory.
Efficient Reinforcement Learning for Reversi AI
Published by
Proc. of The International Conference on Statistics, Applied Mathematics and Computing Science
Summary
Supervised a M.S. student to apply the MAB theory to optimize the game performance.
Frequentist Multi-Armed Bandits for Complex Reward Models
Published by
Proc. of The International Conference on Statistics, Applied Mathematics and Computing Science
Summary
Supervised a M.S. student to extend the frequentist MAB to high dimensions.
Whittle Index for Restless Bandits with Expanding State Spaces
Published by
Numerical Mathematics: A Journal of Chinese Universities
Summary
Extension of J8 to the general infinite state space while establishing a theoretical framework for optimizations.
Dynamic Probing for Intrusion Detection under Resource Constraints
Published by
Proc. of IEEE International Conference on Communications (ICC)
Summary
Cybersecurity optimization and algorithm design: discrete combinatorial models.
Dynamic Intrusion Detection in Resource-Constrained Cyber Networks
Published by
Proc. of IEEE International Symposium on Information Theory (ISIT)
Summary
Cybersecurity optimization and algorithm design: stochastic learning.
Adaptive Shortest-Path Routing under Unknown and Stochastically Varying Link States
Published by
Proc. of the 10th Intl. Symposium on Modeling and Optimization in. Mobile, Ad Hoc, and Wireless Networks (WiOpt)
Summary
Shortest-path algorithms for time-varying stochastic graphs.
Cooperative Game in Dynamic Spectrum Access with Unknown Model and Imperfect Sensing
Published by
IEEE Transactions on Wireless Communications
Summary
Extension of J9 to a communication network with complex noises.
Online Learning for Stochastic Linear Optimization Problems
Published by
Proc. of Information Theory and Applications Workshop (ITA)
Summary
Stochastic linear programming: optimal algorithms.
Indexability and Whittle Index for Restless Bandit Problems Involving Reset Processes
Published by
Proc. of the 50th IEEE Conference on Decision and Control (CDC)
Summary
Collaboration with Prof. Richard Weber from the University of Cambridge.
Multi-Armed Bandit Problems with Heavy-Tailed Reward Distributions
Published by
Proc. of Allerton Conference on Communications, Control, and Computing
Summary
Optimal solution to MAB under complex reward models.
Distributed Learning Under Imperfect Sensing in Cognitive Radio Networks
Published by
Proc. of the 44th Asilomar Conference on Signals, Systems, and Computers
Summary
Online learning and distributed control for cognitive radio networks: noise models.
Distributed Learning in Multi-Armed Bandit with Multiple Players
Published by
IEEE Transactions on Signal Processing
Summary
Extension of game theory to solving distributed frequentist multi-armed bandits while proposing an efficient algorithmic framework for optimizing network performance, with applications in network decentralization and security enhancement.
Indexability of Restless Bandit Problems and Optimality of Whittle Index for Dynamic Multichannel Access
Published by
IEEE Transactions on Information Theory
Summary
Solution to a significant class of online decision problems, referred to as the Bayesian restless multi-armed bandits, reducing the complexity to P with applications in autonomous systems and cognitive communication networks.
Learning from Collisions in Cognitive Radio Networks: Time Division Fair Sharing Without Pre-Agreement
Published by
Proc. of IEEE Military Communication Conference (MILCOM)
Summary
Online learning and distributed control for cognitive radio networks: data models.
Decentralized Multi-Armed Bandit with Imperfect Observations
Published by
Proc. of the 48th Annual Allerton Conference on Communication, Control, and Computing
Summary
Optimal algorithms for distributed MAB under noisy observations.
On Multi-Armed Bandit in Dynamic Systems
Published by
University of California at Davis
Summary
First mathematical completion of the Bayesian and frequentist MAB theory.
Dynamic Multichannel Access with Imperfect Channel State Detection
Published by
IEEE Transactions on Signal Processing
Summary
Optimal protocol design and throughput analysis for a class of stochastic computing networks.
Distributed Learning in Cognitive Radio Networks: Multi-Armed Bandit with Distributed Multiple Players
Published by
Proc. of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)
Summary
Optimal algorithms for distributed MAB in cognitive radio networks.
Decentralized Multi-Armed Bandit with Distributed Multiple Players
Published by
Proc. of the 2010 Information Theory and Applications Workshop (ITA)
Summary
First proposed the distributed MAB model and an algorithmic framework.
On the Myopic Policy for a Class of Restless Bandit Problems with Applications in Dynamic Multichannel Access
Published by
Proc. of the 48th IEEE Conference on Decision and Control (CDC)
Summary
Bayesian MAB in cognitive radio networks: modeling and algorithm design.
Opportunistic Spectrum Access in Self Similar Primary Traffic
Published by
Proc. of IEEE Military Communication Conference (MILCOM)
Summary
Cognitive radio network modeling and control: individual channel selections.
Channel Probing for Opportunistic Access with Multi-Channel Sensing
Published by
Proc. of IEEE Asilomar Conference on Signals, Systems, and Computers
Summary
Cognitive radio network modeling and control: multiple channel selections.
Distributed Sensing and Access in Cognitive Radio Networks
Published by
Proc. of 10th International Symposium on Spread Spectrum Techniques and Applications (ISSSTA)
Summary
Distributed sensors and data transmitters in cognitive radio networks.
A Restless Bandit Formulation of Opportunistic Access: Indexability and Index Policy
Published by
Proc. of IEEE Workshop on Networking Technologies for Software Defined Radio (SDR) Networks
Summary
First mathematical model for cognitive radio network by MAB.
Link Throughput of Multi-Channel Opportunistic Access with Limited Sensing
Published by
Proc. of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)
Summary
Throughput analysis for cognitive radio networks.
Languages
English
Chinese
Skills
Mathematics & Theoretical Foundations
Number Theory, Stochastic Optimization, Reinforcement Learning, Operations Research, Multi-Armed Bandits (MAB), Game Theory, Statistical Optimization, Linear Algebra, Calculus, Probability, Statistics, Mathematical Theory for AI.
Artificial Intelligence & Machine Learning
AI Algorithms, Machine Learning, Deep Neural Networks (DNN), Algorithmic Design, AI E-Commerce, Reversi AI, Data Modeling.
Software & Systems Engineering
Software Design, Algorithm Development, Chip-Manufacturing Systems, Autonomous Systems, High-Accuracy Navigation Systems, System Architecture, Real-time Data Processing, Sensor Fusion, Complex Systems.
Research & Academia
Academic Research, Curriculum Development, Higher Education Teaching, Mentorship, Doctoral Supervision, Scientific Publication, Conference Presentation, Peer Review, Grant Acquisition.
Leadership & Strategy
R&D Leadership, Technical Direction, Strategic Planning, Cross-functional Collaboration, Team Management, Project Management, Innovation Management.